literature review on advertising effectiveness

• Search Menu
• Themed Collections
• Editor's Choice
• Ilona Kickbusch Award
• Supplements
• Author Guidelines
• Submission Online
• Open Access Option
• Self-Archiving Policy
• About Health Promotion International
• Editorial Board
• Advertising and Corporate Services
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

• INTRODUCTION
• SUPPLEMENTARY MATERIAL
• < Previous

Advertising appeals effectiveness: a systematic literature review

• Article contents
• Figures & tables
• Supplementary Data

Murooj Yousef, Sharyn Rundle-Thiele, Timo Dietrich, Advertising appeals effectiveness: a systematic literature review, Health Promotion International , Volume 38, Issue 4, August 2023, daab204, https://doi.org/10.1093/heapro/daab204

• Permissions Icon Permissions

Positive, negative and coactive appeals are used in advertising. The evidence base indicates mixed results making practitioner guidance on optimal advertising appeals difficult. This study aims to identify the most effective advertising appeals and it seeks to synthesize relevant literature up to August 2019. Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses framework a total of 31 studies were identified and analyzed. Emotional appeals, theory utilization, materials, results and quality were examined. Across multiple contexts, results from this review found that positive appeals were more often effective than coactive and negative appeals. Most studies examined fear and humour appeals, reflecting a literature skew towards the two emotional appeals. The Effective Public Health Practice Project framework was applied to assess the quality of the studies and identified that there remains opportunity for improvement in research design of advertising studies. Only one-third of studies utilized theory, signalling the need for more theory testing and application in future research. Scholars should look at increasing methodological strength by drawing more representative samples, establishing strong study designs and valid data collection methods. In the meantime, advertisers are encouraged to employ and test more positive and coactive advertising appeals.

Email alerts

Citing articles via.

• Recommend to Your Librarian
• Journals Career Network

Affiliations

• Online ISSN 1460-2245
• Print ISSN 0957-4824
• Copyright © 2024 Oxford University Press
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Int J Environ Res Public Health

Social Advertising Effectiveness in Driving Action: A Study of Positive, Negative and Coactive Appeals on Social Media

Associated data.

Data available upon request.

Background: Social media offers a cost-effective and wide-reaching advertising platform for marketers. Objectively testing the effectiveness of social media advertising remains difficult due to a lack of guiding frameworks and applicable behavioral measures. This study examines advertising appeals’ effectiveness in driving engagement and actions on and beyond social media platforms. Method: In an experiment, positive, negative and coactive ads were shared on social media and promoted for a week. The three ads were controlled in an A/B testing experiment to ensure applicable comparison. Measures used included impressions, likes, shares and clicks following the multi-actor social media engagement framework. Data were extracted using Facebook ads manager and website data. Significance was tested through a series of chi-square tests. Results: The promoted ads reached over 21,000 users. Significant effect was found for appeal type on engagement and behavioral actions. The findings support the use of negative advertising appeals over positive and coactive appeals. Conclusion: Practically, in the charity and environment context, advertisers aiming to drive engagement on social media as well as behavioral actions beyond social media should consider negative advertising appeals. Theoretically, this study demonstrates the value of using the multi-actor social media engagement framework to test advertising appeal effectiveness. Further, this study proposes an extension to evaluate behavioral outcomes.

1. Introduction

The popularity of social media is growing with advertisers utilizing different platforms to drive online and offline customer engagement [ 1 , 2 ]. As the third largest advertising channel, social media accounts for 13% of global advertising spending [ 3 ]. In 2019, Australian brands spent AUD 2.4 billion on social media advertisements, making social media the second highest expenditure category in digital advertising spending after paid search [ 4 ]. Since the COVID-19 pandemic erupted across the world, social media played a crucial role in disseminating information. Research found social media to be the most rapid digital tool in spreading information regarding the virus, which helped reach and educate specific audiences, such as front-line workers [ 5 ]. With some drawbacks due to the publication of misinformative facts and knowledge, social media platforms remain a major communication platform for scientists, organizations and governments to reach different audience groups and create highly persuasive outcomes [ 6 ]. Similarly, advertisers invest in social media platforms, seeking attention, engagement and action in online and offline, making a clear understanding of social media advertising’s effectiveness of paramount importance.

It is established that emotional appeal messages perform better on social media than rational appeals. Evidence suggests emotional appeals are more likely to achieve engagement and virality [ 1 , 7 ]. However, less is known about the effectiveness of positive, negative and coactive emotional appeals, with the best approach to take remaining unresolved. Recent studies have explored positive versus negative appeals’ effectiveness, with mixed and inconsistent results [ 8 , 9 ]. In fact, examination of coactive appeals has been neglected in contrast to efforts directed at examining positive and negative appeals, further limiting practitioner guidance. This study aims to empirically test the effectiveness of three advertising appeals (i.e., positive, negative and coactive) delivered via social media. Effectiveness is evaluated using online engagement [ 10 ] and behavioral actions.

Social Media Advertising

When creating social media ads, advertisers reportedly prioritize online engagement and utilize this measure to evaluate social media advertising success [ 11 , 12 ]. Quantitative online engagement metrics that social media platforms provide such as likes, comments, shares and clicks are key measures of online engagement. While such metrics provide insights into online engagement, they do not identify actual behaviors taken in response to viewed social media advertisements. To compensate for this omission, scholars have utilized separate data collection tools (i.e., surveys) to evaluate social media effectiveness, relying on consumer self-reports of actions taken following exposure to an advertisement [ 1 ]. The relationship between online engagement (e.g., likes, comments and shares) and behavior (e.g., donations) has received less attention. Therefore, there is limited understanding of social media advertising’s effect on actual behavior. Further, the relationship between online engagement and behavior is not reflected in social media marketing models [ 10 ] and models derived from empirical-based evaluation are needed to address this gap. To this end, the current study investigates social media advertising appeals, comparing three appeals’ (negative, positive and coactive) effects on online engagement and prosocial behavior. Drawing on empirical results, this paper proposes an extension to the social media multi-actor engagement framework outlined by Shawky, Kubacki, Dietrich and Weaven [ 10 ] linking online engagement to behavior. The current study’s focus on social media advertising effectiveness is important theoretically and practically. Theoretically, the current study extends social media evaluation frameworks linking online engagement metrics with behavioral actions. On a practical level, the study enables advertisers to create effective behavior change messages on social media that go beyond likes, comments and shares, delivering empirical evidence outlining the most effective approach to engage audiences and drive action.

2. Literature Review

Advertisements are designed with the ultimate goal of changing behavior [ 13 ]. Commercially, advertisers aim to increase sales by encouraging customers to purchase certain products or choose specific brands [ 8 , 14 , 15 ]. Beyond commercial application, the power of advertising is harnessed to positively change people’s lives, encouraging social and health behaviors such as quitting smoking [ 14 ], encouraging healthy eating [ 15 ], preventing diseases [ 16 ], safe driving [ 17 ] and increasing charity donations [ 18 ]. Such efforts are known as social advertising, the use of promotion and communication techniques to change social behavior [ 19 ]. To motivate the adoption of positive social behavior, social advertising raises awareness, induces action and reinforces maintenance of prosocial behaviors [ 19 , 20 ]. One area where social advertisers may deliver change is the impact of low-quality donations on charities and the environment. One of the most challenging tasks for charities is filtering donations received based on their quality before moving donated goods for redistribution or sale to generate revenue to support essential charity service provision [ 21 ]. Australian charity organizations spend millions of dollars each year on donation sorting processes, ensuring that unusable items donated to charities are discarded and others are remanufactured, while the remaining goods are distributed or sold. In 2018, over USD 9 million was spent sending unusable donations to landfill [ 22 ]. Processing of waste by charities diverts funding away from the delivery of essential community services [ 21 ]. As much as 30% of goods donated are estimated to be unusable, suggesting that there is substantial room for improvement. Despite the magnitude of the problem, limited research focused on improving the quality of donated items is available [ 23 ].

2.1. Emotional Advertising Appeals

As advertisers increasingly seek greater communication effectiveness, the choice of advertising appeal requires more consideration and careful assessment [ 24 ]. Viewers may utilize cognitive or affective evaluation systems when processing an advertisement message [ 25 ]. Rational appeals rely on cognitive evaluations through the persuasive power of arguments or reason to change audience beliefs, attitudes and actions. Such messages are evident in the dissemination of scientific information such as the ones seen during the COVID-19 pandemic. These include facts, infographics and arguments that appeal to a person’s rational processes [ 5 ]. Conversely, emotional appeals utilize the affective evaluation system by evoking emotions to drive action. Recent meta-analytic studies identified that consumers respond more favorably to emotional appeals compared to rational appeals [ 9 , 26 ].

Emotions are defined in many different ways in the literature. In an effort to summarize all definitions in one, Kleinginna and Kleinginna [ 27 ] provide a unified definition that is now relied on by psychology, marketing and other disciplines. They define emotions as “a complex set of interactions among subjective and objective factors, mediated by neural/hormonal systems, which can (a) give rise to affective feelings of arousal, pleasure/displeasure; (b) generate cognitive processes such as emotionally relevant perceptual effects, appraisals, labeling processes; (c) activate widespread physiological adjustments to the arousing conditions; and (d) lead to behavior that is often, but not always, expressive, goal-directed, and adaptive (p. 371)”. For decades, scholars have been studying the effect of emotion on behavior through multiple disciplines and contexts. In the past 10 years, considerable advancement has been clear in research around emotions. Specifically, technology advancements along with new research methodologies allowed scholars to measure and track emotion effects more accurately than before. For example, autonomic measures, including facial expression, heart rate and skin conductance, enabled researchers and practitioners to test emotional responses to certain stimuli [ 28 ]. This, along with digital and social media growth over recent decades, created an opportunity for advertisers to create, manipulate and test different advertising strategies to achieve the highest persuasion effects.

Researchers from psychology, social sciences, health and marketing agree on the crucial role emotions play in shaping human behavior [ 29 , 30 , 31 ]. Emotions have been part of persuasion models as early as the AIDA model, with desire indicating an emotional reaction following the cognition level of attention and interest [ 32 ]. More recently, emotions were found to dominate cognition in a persuasion process, occurring before any cognitive assessment of the message [ 33 ]. Hence, emotions are crucial in advertising’s ability to influence behavior [ 30 , 31 ].

For years, classifying emotions has been a research interest with multiple schools of thought. There are two main ways of classifying emotions, categorically (i.e., discrete emotions) or dimensionally. The discrete emotions approach posits that emotions are specific and defined. Different scholars present different sets of discrete emotions. For example, Ekman [ 34 ] presented six basic emotions, namely anger, disgust, fear, joy, sadness and surprise. Plutchik [ 35 ] argued there are eight basic emotions (fear, anger, sorrow, joy, disgust, acceptance, anticipation and surprise), with mixed emotions producing a secondary emotion (e.g., anger and disgust produce hostility). Models based on the discrete emotions approach appeared, mapping emotions on different dimensions of valence and arousal [ 36 ]. On the other hand, the dimensional theory of emotions classifies emotions based on three main dimensions: (a) valance, (b) arousal and (c) dominance. Hence, emotions can be positive or negative, highly aroused or calm and dominating or under control. Application of the dimensional theory is seen in testing different emotional appeals in advertising with positive emotional appeals and negative emotional appeals, and more recently a mixture of both valanced appeals (i.e., coactive appeals) [ 1 , 7 , 37 ]. The dimensional theory allows for valid comparison of different advertising strategies and appeals and has proven to be valid in multiple empirical results [ 38 , 39 , 40 ]. Hence, the current study employs the dimensional theory of emotions in classifying emotional appeals.

Based on the dimensional theory of emotions, people can perceive any emotional appeal stimulus as pleasant, unpleasant or a mixture of both (i.e., coactive state) [ 41 ]. Hence, emotional appeals are categorized as positive, negative and coactive based on the valance of employed emotions. Emotional appeals research employing the dimensional theory of emotions focuses on the effect of different valanced emotions on cognitive and behavioral actions [ 42 ]. While there is a strong connection between emotional appeals and behavior change [ 43 ], inconsistent results are evident in the literature when comparing positive, negative and coactive appeals (e.g., [ 7 , 8 , 17 , 44 , 45 ]).

2.2. Positive, Negative and Coactive Appeals

While positive emotional appeals were found to increase an individual’s tendency to take action and yield higher message liking [ 9 , 46 ], they are explored and utilized to a lesser extent when contrasted with negative emotional appeals [ 47 , 48 , 49 ]. When positive appeals are studied, humor appeals remain the focus, with less attention directed to the utilization of other positive emotional appeals which may deliver behavioral change [ 9 ]. A review of the literature indicates that positive appeals hold a persuasive advantage in both social and commercial behavior. Wang et al. [ 50 ] found positive admiration appeals to increase purchase intentions more than negative appeals. Similarly, Vaala et al. [ 51 ] support the use of positive empowering appeals when targeting health-related behavior. Positive appeals are especially effective when targeting males [ 52 ], however, studies of positive appeal effectiveness remain limited in number and in execution [ 47 , 48 ]. Some limitations in positive appeals are discussed in the literature. Segev and Fernandes [ 53 ] found positive appeals to be only effective when the behavior requires low effort. Hence, when environmental or climate change advertisements encourage green consumption, recycling or other complex behaviors, positive appeals might be less effective. Similarly, when positive appeals are used to evoke hope in audiences, hope for change is reported in the viewers instead of action taken towards change, indicating an emotion-focused coping function [ 54 ]. Hence, social advertisers remain reluctant to apply positive approaches. Fewer examples of positive appeals being applied to address social issues such as alcohol consumption [ 55 ], obesity [ 56 ], the environment [ 52 ] and safe driving behaviors [ 17 ] are evident.

Negative appeals, on the other hand, dominate research and practice, with over 70% of social advertisements employing negative appeals [ 48 ]. As the main driver of psychic discomfort, negative appeals are utilized to create emotional imbalance to stimulate behavior change [ 57 ]. According to this view, a message that is negatively framed when aiming to drive donations to charities is designed to make the individual feel uncomfortable as they are blamed for the poverty of certain groups (e.g., homeless children). To eliminate such feelings, a viewer is then more likely to contribute to the solution by donating to the charity [ 58 ]. While the use of negative appeals has been found to be effective in multiple contexts, such as healthy eating [ 56 ], moderate alcohol consumption [ 55 ] and safe driving [ 17 ], certain limitations apply. Negative appeals result in developing a coping mechanism such as ignoring the message (i.e., flight) or rejecting the message (i.e., fight), reducing message effectiveness [ 30 , 31 , 59 ]. Furthermore, negative appeals dominate social advertising efforts [ 47 , 48 ], resulting in desensitization to negative emotions, potentially causing such appeals to become less effective [ 57 ]. Finally, negative appeals can serve to reinforce stereotypes, further stigmatizing some people which can lead to reactance in some areas of the community [ 16 ].

Appeals utilizing both positive and negative emotions are labeled inconsistently in the literature. For example, Hong and Lee [ 60 ] and Taute et al. [ 61 ] employ the term mixed emotional appeals while others utilized the term coactive appeal [ 7 , 8 , 62 , 63 , 64 ]. The current study employs the term coactive appeals as coactivity is used to explain the mixed state of emotions and is applied more heavily in the marketing communication literature [ 8 , 62 , 64 ]. When comparing single appeals with coactive appeals that feature an emotional shift (e.g., from positive to negative), coactive appeals were found to be more effective [ 65 , 66 ]. Hence, coactive appeals have recently gained research attention, with advertising studies including such appeals in their evaluations [ 7 ]. Coactive emotional appeals seek to induce both positive and negative emotions simultaneously or as a flow from one appeal to the other [ 42 , 49 ]. For example, a coactive message can take the viewer on an emotional journey either from negative to positive or from positive to negative. The use of a negative to positive emotional flow or a threat–relief emotional message is hypothesized to result in a stronger persuasion outcome [ 49 ]. A recent study by Gebreselassie Andinet and Bougie [ 67 ] found the flow from negative to positive appeals produced more desirable results than negative or positive appeals alone. Similarly, Rossiter and Thornton [ 68 ] found fear–relief appeals to reduce young adults’ speed choice when driving. This is due to positive appeals’ ability to reduce different defensive reactions (e.g., fight, flight) that negative appeals generate. When a positive appeal is added to a negative appeal, post-exposure discomfort is reduced, resulting in the combination of appeals (i.e., coactive) being more effective in changing behavior [ 66 , 69 , 70 ]. Nonetheless, negative appeals remain highly featured in social advertising messages. This is attributed to the rich action tendency potential negative appeals hold [ 71 ], along with their ability to activate the brain more than other emotions [ 72 ]. Negative appeals have the ability to drive action without being liked first. This explains their dominance in social advertising messages and heavy focus in the literature. No known study has empirically tested and contrasted the effectiveness of a coactive appeal with positive and negative appeals directly on social media platforms. Previously studies compared the three appeals using self-report data collection measures, an approach that is limited by social desirability effects [ 73 ]. This study eliminates such limitations by utilizing social media advertising tools and measures where data are collected based on the viewer’s actual reactions on the platform (e.g., likes and clicks) rather than intentions to perform such reactions [ 1 , 74 ].

2.3. Theory

This study applies and builds on the multi-actor engagement framework proposed by Shawky, Kubacki, Dietrich and Weaven [ 10 ]. The framework provides an “integrated, dynamic and measurable framework for managing customer engagement on social media” enabling marketers to understand the different levels of engagement and measure the success of their campaigns [ 10 ]. As social media grows beyond simple dyadic exchanges between customers and companies, the multi-actor engagement framework operationalizes the different levels of engagement in a multi-actor ecosystem where customers, fans, organizations and stakeholders all contribute to levels of engagement with content. The different levels of engagement set by Shawky, Kubacki, Dietrich and Weaven [ 10 ] include connection, interaction, loyalty and advocacy. Connection is defined as a one-way communication where content is presented to customers without any action taken by the customer. When the stimulus attracts a customer’s attention, connection is achieved. In other words, Schivinski et al. [ 75 ] label this level as the consumption stage where social media users consume content but do not necessarily interact with it. At this level, customers passively consume the online content without taking any action yet [ 75 , 76 ]. Based on the Shawky, Kubacki, Dietrich and Weaven [ 10 ] framework, this level is measured by reach and impressions. Reach is defined as the number of unique users who viewed an advertisement, while impressions are recorded every time an ad is viewed, including multiple views by the same user [ 77 ]. The next level in the Shawky, Kubacki, Dietrich and Weaven [ 10 ] framework is interaction, and this level highlights the beginning of two-way communication between different actors, including customers, organizations and other customers. At this level, users engage with the advertisement by interactively contributing to the advertised message [ 75 ]. Social media interaction can be defined as “the number of participant interactions stratified by interaction type” [ 78 ]. Interaction types include likes, comments, clicks and overall engagement which are utilized to measure this level following the Shawky, Kubacki, Dietrich and Weaven [ 10 ] framework. The third level in the Shawky, Kubacki, Dietrich and Weaven [ 10 ] framework is loyalty, where interaction is repeated over time. A user is regarded as loyal if they are consistently seen interacting and contributing to an organization’s advertisements and content on social media. To encourage loyalty on social media, an organization’s content should aspire to complement the user’s image, as this will increase the chances of interactions over time and sharing with others [ 79 ]. This level is measured by multiple comments and messages in the Shawky, Kubacki, Dietrich and Weaven [ 10 ] framework. Finally, advocacy marks the fourth and highest level of engagement. Advocacy is recorded when customers spread an organization’s message by generating new content through their networks. Advocacy is where interaction is sustained, and support moves beyond the dyadic nature, reaching users’ own networks. This is when users share organizations’ content with their community circles through their own pages, profiles and networks [ 80 ]. Following the Shawky, Kubacki, Dietrich and Weaven [ 10 ] framework, this level is measured by shares, tagging of others on a post and word of mouth. The last two levels are parallel to Schivinski, Christodoulides and Dabrowski [ 75 ] creation level where users generate and create content. This is regarded as the highest level of online engagement as it motivates future interaction and involvement with the organization online and offline [ 76 ].

The multi-actor engagement framework stops at advocacy as the highest level of engagement. As a result of our study, we propose a fifth level which marks the transition from online engagement to behavior. The fifth level signifies the customer’s action beyond social media platforms, and this can be reflected by purchases, donations, registrations, signatures on a petition and many other actions. While previous studies explore advertising effectiveness with customer perceptions such as attitudes, memorability, likability of the ad and intentions to take action, the direct effect of advertising on behavior, specifically in a social media context, is yet to be explored [ 71 ]. This is now possible with methodological advances and digital and social media platforms that allow for experiments to track not only automated measures on social media (e.g., likes and shares), but further action taken beyond such platforms (e.g., filling lead form, visiting a store, buying a product) [ 71 ]. This is of specific interest to advertisers employing emotional appeals, as each emotion has different action tendencies which influence the audience behavior after being exposed to the emotional advertisement appeal. As Poels and Dewitte [ 71 ] explain, different emotional appeals “help the individual sort out which action tendency is the most functional in this situation”. Each behavior is tracked differently, for example, weight loss campaigns can be tracked through the audience’s eating habits and exercise patterns, while antitobacco campaigns may track cigarette purchases, hence, this level has a number of possible measures. If measuring purchases of a product or donations to an online charity, the click through rate to the product or the donation page along with the number of orders or the amount of donations are examples of measures that reflect actions. For the purpose of this study, we measure actions through the number of requests to receive a donation sorting bag that helps reduce textile waste. The extended model is shown in Figure 1 .

The extended multi-actor engagement framework [ 10 ].

Past literature identified that positive advertising appeals produce an emotion-focused coping mechanism, while negative appeals create emotional imbalance to stimulate behavior change, and coactive appeals may reduce defensive reactions with less evidence of effectiveness in creating behavior change. For the purpose of the current study, we focus on four online behavioral outcomes. First, connection is defined by reach. Second, interaction is defined by engagement, likes, comments and clicks. Third, loyalty is defined by repeated actions on the ad. Finally, advocacy is defined by sharing the ad. Guided by past research, the current study expects negative advertising appeals to be more effective in evoking online behavioral responses than positive and coactive message on social media. Hence, the following hypotheses are proposed:

Negative advertising appeals will achieve more interactions than positive and coactive advertising appeals on social media .

Negative advertising appeals will achieve more loyalty than positive and coactive advertising appeals on social media .

Negative advertising appeals will achieve more advocacy than positive and coactive advertising appeals on social media .

Negative advertising appeals will achieve more behavior actions than positive and coactive advertising appeals on social media .

2.4. Gaps and Aims

This study aims to address three main gaps in the literature. First, the need for a social media advertisement evaluation model that addresses both online engagement as well as behavior actions which is addressed through an empirical study. Second, the evaluation of social media advertisements has been limited by self-reported measures of intentions to engage with advertisements (e.g., intention to click, like, share) [ 81 ], neglecting actual engagement measures (e.g., comments, reactions, shares, likes and ad clicks) that can be directly observed in social media. Third, current evidence on social media advertising appeals’ effectiveness in engagement and changing behavior remains conflicted, inconsistent and fragmented. Such gaps create a challenge for researchers aiming to understand social media advertising appeal effectiveness and advertisers, given that limited guidance is available to provide an implementation roadmap that can be relied upon to deliver behavior change benefitting people. This study addresses the aforementioned gaps testing the capacity of positive, negative and coactive advertising appeals to engage audiences on social media and drive behavioral actions.

3. Material and Methods

The current study employed an experimental study design, where three advertising appeals (positive, negative and coactive) were designed following Alhabash, McAlister, Hagerstrom, Quilliam, Rifon and Richards [ 7 ] and Hong and Lee [ 60 ] and published on Facebook following a pre-test conducted with a participant panel ( n = 10).

3.1. Pre-Test

An online survey was distributed featuring the three advertisements. After exposure to each ad, participants were asked to rate how the advertisement made them feel on a 7-point scale (mostly positive/mostly negative) [ 82 ]. Next, participants were asked to describe how the advertisements made them feel in one word. The three ads maintained similarities in visuals and manipulated verbal elements to represent each appeal. The pre-test included one between-subjects ANOVA to compare advertisements’ emotional valance and a sentiment analysis of each advertisement response. The aim was to ensure that the positive advertisements were rated as more positive than the negative and coactive advertisements, the negative advertisements more negative than positive and coactive advertisements and the coactive advertisements in the middle. Moreover, the pre-test included a sentiment analysis of participants’ feedback on each advertisement. Word maps were generated and analyzed to confirm each ad represented the respective appeal.

3.2. Social Media Advertisements

After the pre-test, the three advertisements were published on Facebook and promoted for a week, controlling for the reach (i.e., number of people who were presented with the ad) of each advertisement through Facebook’s A/B testing tool on Facebook ads manager. The A/B testing tool allows for a comparable data set between the tested advertisements by controlling for reach across the different groups along with demographic elements (e.g., gender). Facebook ads manager allows for extraction of advertisements’ performance data as a spreadsheet, which was then used by the research team to analyze advertising appeal effectiveness using SPSS v.25. Facebook records advertisements’ performance data in key metrics including reach, likes, comments and shares. The ad appeared to Facebook users on their news feed as they scrolled through the content. The published ads (see Table 1 ) were linked to a charity website. One aim of the website is to educate people on what to donate to increase the quality of donations for Australian charities. When landing on the website, customers were asked to fill in a form to request a cloth sorting bag for their donations. Each form submission was recorded, and web data were extracted for all form submissions when the campaign was over. The research procedure is outlined in Figure 2 .

Research procedure.

Overview of employed stimulus.

3.3. Analysis and Measures

The three advertisements employed in this study were analyzed using Shawky, Kubacki, Dietrich and Weaven [ 10 ] multi-actor social media engagement framework. When customers viewed the advertisement, reach was recorded. When a customer liked or reacted to or clicked on an ad, interaction was recorded. When customers commented multiple times or replied to others to clarify the message or provide information, loyalty was recorded. When users shared the advertisement, advocacy was recorded. Finally, when customers filled in the form on the charity website, action was recorded. The form was created as a lead capture tool where customers filled in their information (e.g., name, address, contact details) to request a donation bag they could use to take their donations to charities. Three separate forms were created with three links for each advertising appeal. Data of each ad’s performance were extracted from Facebook ads manager while data of all request forms were extracted from the website after the ads on Facebook ended and were analyzed based on the number of requests received on each form. Following Merchant, Weibel, Patrick, Fowler, Norman, Gupta, Servetas, Calfas, Raste, Pina, Donohue, Griswold and Marshall [ 78 ], the data received were analyzed as categorical (out of all users reached, ad was liked: yes or no, ad was clicked on: yes or no) and continuous in the sense how many liked, clicked, commented. A chi-square test of independence was performed to examine the relation between advertising appeal and Shawky, Kubacki, Dietrich and Weaven [ 10 ] engagement levels: connection, interaction, loyalty, advocacy and the fifth proposed level of behavior, using SPSS v.25.

A sample of ten participants was achieved for the pre-test with a mean age of 24 and balanced gender (50% females). Using SPSS v.25, pre-tests were successful for all advertising appeals. There was a statistically significant difference between group means showing a significant effect of appeal type on emotional valence (mostly positive/mostly negative) at the p < 0.05 level as determined by one-way ANOVA (F(2,27) = 199.957, p = 0.00). Post hoc analyses were conducted using Tukey’s post hoc test. The test showed that the three advertising appeal groups differed significantly at p < 0.05. The positive appeal ad (M = 6.60, SD = 0.52) was significantly more positive than the negative ad (M = 1.40, SD = 0.51) and the coactive ad (M = 4.40, SD = 0.69). Similarly, the negative ad (M = 1.40, SD = 0.51) was significantly more negative than the positive (M = 6.60, SD = 0.52) and coactive ads (M = 4.40, SD = 0.69). The coactive ad means appear in the middle as their mean scores were mostly neutral compared to the other two categories of emotional tone (see Figure 3 ). This result indicated that in the coactive condition, participants perceived the advertisement as both positive and negative at the same time, reflecting the bi-dimensional nature of the appeal (negative and positive).

Pre-test mean scores of emotional valences.

The sentiment analysis showed that the positive appeal was perceived as mostly hopeful, the negative mostly shameful and the coactive was perceived as motivational. Figure 4 showcases the word map for each advertisement.

Word map based on sentiment analysis.

4.1. Social Media Advertisements

The three promoted appeals achieved a total of 23,905 impressions and reached 21,054 users which resulted in 787 clicks to the website. Facebook ads manager targeted a balanced sample for the three promoted appeals by using its A/B testing tool. The three ads reached Facebook users above 18 years of age of both genders (see Figure 5 and Figure 6 ). While the overall sample is female skewed (see Figure 6 ), each advertising appeal achieved a balanced reach for both genders (see Figure 7 ). The click through rate achieved through the three ads of 3.28% is considered above the average of 1.24% for Facebook ads [ 83 ]. A total of 28 requests were received for donation bags through the website forms. The results for each advertising appeal are discussed next.

Gender distribution across the three ads.

Gender distribution across advertisements.

Distribution of females across the three appeals.

4.1.1. Connection

Connection was measured through reach and was controlled between the three appeal ads to ensure applicable comparison (see Table 2 ). A chi-square test of independence revealed an insignificant effect of appeal type on reach between the three advertisements χ 2 (2, N = 21,054) = 4.57, p = 0.11.

Number of people reached compared to impressions on positive, negative and coactive appeals.

4.1.2. Interaction

Appeal type had a significant effect on the level of interaction. This is evident through all three measures: clicks, engagement and comments (see Table 3 ). The negative appeal ad had significantly more engagement than the positive and coactive appeals. A chi-square test of independence showed significance for clicks χ 2 (2, N = 21,054) = 18.57 p < 0.05, and engagement χ 2 (2, N = 21,054) = 20.68 p < 0.05. No significant difference was observed for comments χ 2 (2, N = 21,054) = 4.94 p < 1, partially supporting H1. When comparing clicks on the positive and coactive appeals, no statistical significance was recorded at the 0.05 level χ 2 (1, N = 13,851) = 1.49 p = 1.11. Similarly, effect was insignificant when comparing engagement χ 2 (1, N = 13,851) = 1.48 p = 1.15 and comments χ 2 (1, N = 13,851) = 1.02 p = 0.3 between positive and coactive appeals.

Number of interactions on positive, negative and coactive appeals.

4.1.3. Loyalty

No repeated interaction was recorded for any of the three advertising appeals (see Table 4 ). Therefore, appeal type had no effect on level of loyalty. Hence, H2 was not supported.

Number of repeated interactions on positive, negative and coactive appeals.

4.1.4. Advocacy

A chi-square test of independence showed no significant effect of appeal type on advocacy (see Table 5 ). This is seen in the number of shares the three appeals received χ 2 (2, N = 21,054) = 1.82 p = 0.40. Hence, H3 was not supported.

Number of shares on positive, negative and coactive appeals.

4.1.5. Behavior

Behavior was measured through the number of requests for a donation sorting bag received for each advertising appeal. The negative appeal achieved the highest number of requests, followed by positive and coactive appeals (see Table 6 ). A chi-square test of independence showed a significant difference for appeal type based on the number of bag requests χ 2 (2, N = 21,054) = 6.54 p < 0.05, supporting H4.

Number of requests on positive, negative and coactive appeals.

5. Discussion

The current study contributes to the literature in three ways. Firstly, we tested positive, negative and coactive appeals’ effectiveness on social media to understand their effect on engagement and behavior. This is the first study to directly examine advertising appeals on social media without the use of self-report measures. Our findings support the use of negative appeals over positive and coactive appeals when aiming to drive engagement and change behavior. This provides clear guidance for practitioners aiming to create effective social advertisement messages on social media. Secondly, this is the first study to apply and build on the Shawky, Kubacki, Dietrich and Weaven [ 10 ] social media multi-actor engagement framework in testing advertising appeals’ effectiveness. Our findings support the use of the framework in testing advertising effectiveness and show clear measures for each level of engagement. Finally, the study proposed an extension to the social media multi-actor engagement framework [ 10 ] with a clear and practical way of measuring actions beyond social media engagement. This will enable social advertisers to measure advertising effectiveness on actual behavior, moving beyond indirect behavioral measures such as attitudes, norms and intentions. Each contribution will be discussed in detail next.

5.1. Negativity Increases Appeals’ Effectiveness

When comparing positive, negative and coactive appeals’ performance in driving engagement and action on social media, our findings suggest negative appeals hold a persuasive advantage (see Figure 8 ). This is evident in the significant increase in engagement and actions for the negative advertisements when compared with the positive and coactive advertising appeals. This is consistent with previous findings, especially with behavior related to charities [ 18 , 84 ] and the environment [ 52 , 85 ]. Our findings support the limited effectiveness of positive appeals when complex issues are discussed. The advertisements employed by the current study address the issue of waste and its impact on the planet, where positive appeals have been found to be less effective in the past [ 54 ].

Comparison of advertising appeal performance.

The effectiveness of coactive appeals has not been tested directly on social media before, marking a significant contribution of this study. Interestingly, the coactive appeal was equally effective when compared to positive appeals in attracting comments, and driving loyalty, advocacy and behavioral actions. The findings in this study indicate that both positive and coactive appeals performed in a similar way, contradicting previous findings supporting positive appeals’ effectiveness over coactive appeals [ 7 ]. The limited effectiveness of both positive and coactive appeals in this study may be attributed to the advertising platform.

5.2. Platform Effect on Appeal Effectiveness

Social media advertising engagement differs across social media platforms [ 86 , 87 ]. To understand why negative appeals were most effective in our experiment, a review of the platform of choice (i.e., Facebook) was necessary. Evidence suggests Facebook is among the most negative platforms in nature. Voorveld, van Noort, Muntinga and Bronner [ 86 ] explain the implications of such findings by relating them to advertising valence (i.e., appeals). Hence, when advertising on Facebook, advertisements that evoke negative feelings (i.e., negative appeals) perform best. The effectiveness of such appeals stems from the fluency between advertising appeal and platform nature [ 87 ]. Platform–appeal fit was found to increase the effectiveness of advertising and drive higher rates of engagement [ 87 ]. In fact, Facebook ran an experiment in 2014, where content was manipulated for half a million users. The experiment tested whether what users share is affected by what they see in their newsfeed. The findings supported the concept of platform–appeal fit. When negative content was increased in users’ feeds, their posts became more negative as a result [ 88 ]. Recently, Facebook was criticized for carrying out such experiments, which can have an impact on mental health, personal decisions and in many cases political election outcomes [ 89 ]. All of this results in users’ skepticism of the platform, contributing to its negative nature.

5.3. Driving Action Beyond Social Media Engagement

The current study applies and builds on the Shawky, Kubacki, Dietrich and Weaven [ 10 ] social media multi- actor engagement framework. The framework provides a practical tool for organizations seeking to evaluate their social media content. This study applies the framework, testing advertising appeals’ effectiveness, an application of the framework that has not previously occurred. It showcases the ability to use the Shawky, Kubacki, Dietrich and Weaven [ 10 ] framework as a tool to evaluate social media advertising effectiveness in driving engagement and prosocial behaviors. While the framework proved to be a practical and easy measurement tool for advertisers, it has a key limitation when aiming to carry out a thorough evaluation of advertising messages’ capacity to drive action. Linking online engagement to behavior remains limited when evaluating social media advertising effectiveness. Hence, an extension to the model is proposed, where behavior is measured through action (e.g., donations), extending understanding beyond social media engagement (see Figure 9 ). This extension could be tested empirically, allowing actions taken to be observed and analyzed. Behavior could be measured as a fifth step in Shawky et al.’s framework.

Proposed extension to Shawky et al. (2020) social media multi-actor engagement framework.

Negative appeals achieved the highest actions, while positive and coactive appeals received equal actions from users. This could be explained by the emotion-focused coping function, where positive emotions produce a hope for change rather than inducing a motivation to take action [ 54 ]. Furthermore, when viewers hold favorable prior attitudes towards the advertised behavior (i.e., reduce waste) positive appeals are found to be less effective [ 90 ]. While there are no data on prior attitudes of the viewers for this experiment, the comments received about the negative appeal advertisement suggest some people are passionate about the environment, and they want to take actions to help others and reduce waste. Hence, negative appeals were more effective in driving both engagement and action.

5.4. Limitations and Future Research

Three main limitations apply to this study. First, mediators of effectiveness, such as prior attitude towards the issue, were not examined in this study. Future research is recommended to employ a pre-exposure survey to collect such data or utilize social media targeting tools to target specific audiences with certain interests. Second, the advertisements tested in this study were shared predominantly on Facebook where negative content dominates. Future research should investigate other platforms to understand the effects that social media platforms exert on advertising appeal effectiveness. Empirical tests of different appeals on multiple platforms (e.g., Twitter, TikTok, Snapchat) are needed to draw conclusions on where different appeals perform best. Third, the sample reached by this study may be small when compared to other studies in social media settings [ 91 ], and future research may increase the reach by increasing the budget invested in the advertising campaign on Facebook.

It is important to note that different contexts may achieve different results, and the experiment can be replicated to understand if other behaviors, such as the uptake of the COVID-19 vaccine, are more effectively achieved through negative advertising appeals. Research shows a promising effect of emotional appeals in both social and health domains, with more empirical evidence needed [ 92 ]. Future research may investigate the role of social media advertisements in inspiring loyalty and advocacy through different emotional appeals. The current study found no effect of appeal type on loyalty and advocacy, presenting a limitation to the overall findings. To increase analytic rigor, future research may employ a CB/PLS-SEM approach to test social media advertising’s effect on behavior [ 93 ].

6. Conclusions

This study examined positive, negative and coactive advertising appeals’ effectiveness in driving engagement and actions on and beyond social media platforms. Findings support the use of negative advertising appeals over positive and coactive appeals. The results highlight how negative appeals on social media advertising in an environmental and charity context can deliver superior outcomes to engage more people and positively impact social behavior. Theoretically, this study highlights the value of using Shawky et al.’s (2020) multi-actor social media engagement framework to test social advertising appeal effectiveness and provides a practical extension to evaluate behavioral outcomes.

Author Contributions

Conceptualization, M.Y., T.D. and S.R.-T.; methodology, M.Y.; software, M.Y. and T.D.; validation, M.Y., S.R.-T. and T.D.; formal analysis, M.Y.; investigation, M.Y.; resources, M.Y.; data curation, M.Y.; writing—original draft preparation, M.Y.; writing—review and editing, M.Y., T.D. and S.R.-T.; visualization, M.Y.; supervision, T.D. and S.R.-T.; project administration, M.Y. and T.D. All authors have read and agreed to the published version of the manuscript.

This research received no external funding.

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by Ethics Committee of Griffith University (protocol code 2019/697 and date of approval 3 September 2019).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

Conflicts of interest.

The authors declare no conflict of interest.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

The impact of interactive advertising on consumer engagement, recall, and understanding: A scoping systematic review for informing regulatory science

Roles Conceptualization, Data curation, Formal analysis, Writing – original draft, Writing – review & editing

Affiliation RTI International, Research Triangle Park, Durham, NC, United States of America

Roles Data curation, Formal analysis, Writing – original draft, Writing – review & editing

Roles Data curation, Writing – review & editing

Roles Conceptualization, Funding acquisition, Writing – review & editing

Affiliation Office of Prescription Drug Promotion, Center for Drug Evaluation and Research, US Food and Drug Administration, Silver Spring, Maryland, United States of America

Roles Conceptualization, Data curation, Formal analysis, Funding acquisition, Project administration, Supervision, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

• Kristen Giombi, 
• Catherine Viator, 
• Juliana Hoover, 
• Janice Tzeng, 
• Helen W. Sullivan, 
• Amie C. O’Donoghue, 
• Brian G. Southwell, 
• Leila C. Kahwati

• Published: February 3, 2022
• https://doi.org/10.1371/journal.pone.0263339
• Reader Comments

We conducted a scoping systematic review with respect to how consumer engagement with interactive advertising is evaluated and if interactive features influence consumer recall, awareness, or comprehension of product claims and risk disclosures for informing regulatory science. MEDLINE, PsycINFO, Business Source Corporate, and SCOPUS were searched for original research published from 1997 through February 2021. Two reviewers independently screened titles/abstracts and full-text articles for inclusion. Outcomes were abstracted into a structured abstraction form. We included 32 studies overall. The types of interactive ads evaluated included website banner and pop up ads, search engine ads, interactive TV ads, advergames, product websites, digital magazine ads, and ads on social network sites. Twenty-three studies reported objective measures of engagement using observational analyses or laboratory-based experiments. In nine studies evaluating the association between different interactivity features and outcomes, the evidence was mixed on whether more interactivity improves or worsens recall and comprehension. Studies vary with respect to populations, designs, ads evaluated, and outcomes assessed.

Citation: Giombi K, Viator C, Hoover J, Tzeng J, Sullivan HW, O’Donoghue AC, et al. (2022) The impact of interactive advertising on consumer engagement, recall, and understanding: A scoping systematic review for informing regulatory science. PLoS ONE 17(2): e0263339. https://doi.org/10.1371/journal.pone.0263339

Editor: Qihong Liu, University of Oklahama Norman Campus: The University of Oklahoma, UNITED STATES

Received: September 15, 2021; Accepted: January 15, 2022; Published: February 3, 2022

This is an open access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.

Data Availability: All relevant data are within the paper and its Supporting Information files.

Funding: Funded through a contract from the U.S. Food and Drug Administration to RTI International (Contract 75F40120A00017, Order Number 75F40120F19003). KG, CV, JH, JT, BS, LK are employees of RTI International. HS and AO are employees of the U.S. Food and Drug Administration. HS and AO (employees of the sponsor) participated in the study design, decision to publish, and critically reviewed the manuscript prior to submission.

Competing interests: HS and AO are employees of the study sponsor. This does not alter our adherence to PLOS ONE policies on sharing data and materials.

1. Introduction

In 2020, it is estimated that nearly $356 billion was spent on digital advertising in the United States [ 1 ]. Much of this advertising consists of display ads, social media ads, search engine marketing, and email marketing often with interactive components to target the 85% of US adults who go online daily [ 2 ]. An interactive ad encourages consumers to interact with the ad (and thus the brand), rather than just passively view the ad. Although interactivity is often considered a vital element of successful online advertising [ 3 , 4 ], its impact on consumer engagement and decision-making is not entirely clear.

The academic definition of interactive advertising has evolved and varied at least in part as possibilities for ad design and placement have shifted, meaning interactive advertising can be defined differently depending on the context. Experts have defined interactive ads in terms of processes, features, and/or user perceptions, and no consensus about the definition has been reached to date [ 5 – 14 ]. Conceptual frameworks considered by researchers in approaching interactive advertising have tended to include descriptions of how users behave in response to ads [ 13 , 15 – 17 ]. Metrics employed by the advertising industry also have shifted over time. The operationalization of interactive advertising often has been determined by the conceptual framework used and the outcome of interest to the researcher.

With an increased presence of interactive advertising in digital and social media [ 18 ], it is critical to understand how consumers engage with these types of advertisements and whether interactive features influence consumer recall, awareness, or comprehension of product claims and risk disclosures. This is of particular importance for products or services for which advertising content is regulated, such as prescription drugs, alcohol, tobacco, and financial products or services, to ensure that such advertising does not introduce barriers or challenges to consumer understanding of risks associated with such products. Especially within the past decade, regulatory science researchers have embraced the tools of social science to assess consumer perceptions of risk as well as potential impediments to consumer understanding [ 19 , 20 ]. Social science research can offer evidence of advertising effects on consumer perceptions, and agencies such as the U.S. Food and Drug Administration have used such approaches to assess consumer engagement with different types of advertisements, such as direct-to-consumer prescription drug television ads [ 21 ]. In order to assess whether interactive advertising poses new theoretical challenges or opportunities, we conducted a scoping systematic review to summarize the research related to consumer engagement with interactive advertisements and impact on recall and understanding of product claims and risk disclosures.

The protocol for this scoping review was registered at Open Science Framework on October 26, 2020 [ 22 ]. The goal of this scoping systematic review was to describe the extant literature on interactive advertising and consumer engagement, particularly as it concerned regulated product advertising and its influence on comprehension of product claims and risk disclosures. We designed the four research questions (RQs) that guided this scoping review to identify gaps in the evidence base and summarize important considerations needed to inform the design and conduct of future primary research studies in this area. The four RQs were:

• RQ 1: What methods and measures are used to evaluate consumer engagement with interactive advertisements in empirical studies?
• RQ 2: In empirical studies of interactive advertising in naturalistic or real-world contexts, to what extent do consumers engage with interactive advertisements?
• RQ 3: What is the association between features of interactive advertisements for goods or services and consumer engagement, recall, awareness, or comprehension of product claims and risk disclosures?
• RQ 4: How do interactive advertisements for goods and services compare to non-interactive advertisements (e.g., traditional print or broadcast advertisements) with respect to consumer engagement, recall, awareness, and comprehension of product claims and risk disclosures?

2.1 Search and data sources

We searched MEDLINE via PubMed, PsycINFO, Business Source Corporate, and SCOPUS for original research published in English from January 1, 1997, through February 17, 2021, using search terms related to advertising and marketing, internet, and the outcomes of interest (e.g., engagement, knowledge, click-through rate). Little research on digital advertising was conducted prior to the mid-1990s, and our preliminary evidence scan showed very few papers published prior to 1997. The detailed search strategy is in S1 Appendix . We also searched reference lists of systematic and narrative reviews and editorials where relevant.

2.2 Study selection

Two reviewers independently screened titles/abstracts and full-text articles for inclusion based on study selection criteria for each research question. Disagreements at the full-text review stage were resolved by a third reviewer. Detailed study selection criteria are described in S2 Appendix . In brief, we included all studies among persons of any age in the general public who were characterized as being a potential consumer target for interactive advertising. For all RQs, we included studies that examined exposure to interactive advertisements, which we defined as the promotion of a product, service, or idea using various features or tools that provide the opportunity for persons to interact directly with the ad and potentially influence/inform the remaining sequence, appearance, or content to be presented about the product, service, or idea. For RQ 2, we included only studies with exposure to interactive advertising in naturalistic or real-world contexts. For RQ 3, studies that compared alternative versions of advertisements with interactive elements that varied with respect to the type or level of interactivity were selected. For RQ 4, studies that compared interactive advertisements with traditional advertising (i.e., print ads, broadcast ads, or online/internet ads without interactive elements) were included.

Eligible outcomes varied by RQ. For RQ 1, we included studies with any measure of consumer engagement. For RQ 2, we required objective measures of engagement such as time spent viewing, content navigation, click-through rates, page views, shares, likes, or leaving comments. For RQs 3 and 4, we required studies to report outcomes including consumer recall, awareness, and comprehension of product claims, risk disclosures, or both. Lastly, we included only studies conducted in countries designated as very highly developed per the United Nations Human Development Index to maximize applicability to decision-makers in such settings [ 23 ].

2.3 Data abstraction and synthesis

For each article included, one reviewer abstracted relevant study characteristics and outcomes into a structured abstraction form, and a second senior reviewer checked the form for completeness and accuracy. We narratively synthesized findings for each RQ by summarizing the characteristics and results of the included studies in narrative and tabular formats. Because this was designed as a scoping review, we did not conduct risk of bias assessments on included studies, quantitatively synthesize findings, or conduct strength of evidence assessments.

We screened 3,765 titles and abstracts and 136 full-text articles. We included 32 studies published in 33 articles ( Fig 1 ) [ 7 , 24 – 55 ]. Twenty-three studies addressed RQ 1, eight studies addressed RQ 2, nine studies addressed RQ 3, and four studies addressed RQ 4. An overview of included studies is provided in Table S4-1 in S4 Appendix . A list of full-text studies that we reviewed and excluded is provided in the S3 Appendix .

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pone.0263339.g001

3.1 Research question 1: What methods and measures are used to evaluate consumer engagement with interactive advertisements in empirical studies?

3.1.1 study characteristics..

We identified 23 studies eligible for RQ 1 that were published between the years 1997 and 2019 and conducted across multiple countries [ 24 – 30 , 33 , 35 – 42 , 46 , 47 , 50 – 53 ]. An overview of the studies is in Table S4-1 and S4-2 in S4 Appendix . Six were observational studies evaluating consumer response to real-world advertisements or campaigns [ 24 , 25 , 28 – 30 , 37 ]. The rest of the studies were experiments conducted in laboratory or controlled environments. The sample sizes across the included studies ranged from 20 to 116,168 participants; however, two studies [ 29 , 30 ] did not report the number of persons participating in the study.

The types of interactive advertisements evaluated varied across the included studies. Six studies [ 26 , 33 , 35 , 40 , 47 , 50 ] evaluated banner ads, three studies [ 7 , 36 , 46 ] evaluated product websites, three studies [ 29 , 30 , 41 ] evaluated paid search engine ads, three studies [ 38 , 51 , 52 ] evaluated interactive television ads, two studies [ 24 , 27 ] evaluated social network site ads, one study [ 39 ] evaluated a pop-up ad and the rest of the studies evaluated other types of digital ads. This included short-message-service TV marketing [ 37 ], an ad with a video clip embedded in a digital magazine [ 42 ], ads within a simulated online store [ 53 ], and combinations of different types of digital and online ads [ 25 , 30 ]. The type of products advertised across the included studies included unregulated consumer products (e.g., digital cameras) and services (e.g., travel planning); regulated products and services (car insurance, financial); and health/health behavior campaigns.

3.1.2 Findings.

An overview of findings is in Fig 2 . Authors of the six observational studies reported engagement outcomes associated with real-world advertising or marketing campaigns [ 24 , 25 , 28 – 30 , 37 ]. Authors of four studies reported objective measures of the proportion of users exposed to an ad that clicked on the ad (i.e., “click-through rates”) by using platform-specific (e.g., Facebook, Google AdWords) analytic tools for advertisers [ 24 , 29 ], specialized web tracking software that members of a market research panel consented to have installed on their computers to monitor web behavior [ 28 ], or a unique event identifier created on the advertiser’s server whenever an online ad was clicked [ 30 ]. Authors of the other two observational studies reported subjective measures of engagement. In one study, authors used audio, computer-assisted self-interviews that asked respondents about their engagement with online marketing of a specific class of product [ 25 ]. In the other study, authors used post-campaign surveys (mode not specified) to evaluate engagement outcomes [ 37 ].

https://doi.org/10.1371/journal.pone.0263339.g002

Authors of the 17 experimental studies reported engagement outcomes from experiments using actual real-world ads or from experiments using fictitious ads designed specifically for the experiment. Authors of the experimental studies controlled participant exposure to the ads, and depending on the measure, outcome measurement occurred either concurrently with the ad exposure or through completion of post-exposure surveys or interviews.

Two of the experimental studies used objective measures of ad engagement employing eye-tracking technologies during exposure to evaluate user engagement with digital ads placed on online platforms (Facebook page, blog, and industry-specific search engine) [ 26 , 50 ]. In Muñoz-Leiva, Hernández-Méndez, and Gómez-Carmona [ 26 ] the ads used were fictitious, and the sites they were placed on were mocked up to resemble existing platforms (e.g., Facebook). In Barreto [ 50 ], each participant’s own Facebook page and the authentic Facebook page for a specific brand of athletic shoe was used. In both studies, authors first calibrated the eye-tracking equipment for each participant, then assigned one or more tasks for the participants to complete (e.g., navigate to find a specific item). The eye-tracking technology measured fixation counts and duration of fixation on the ad portion of the screens as participants navigated through the task.

Seven of the experimental studies were designed using a within- or between-subjects randomized factorial design or both [ 27 , 33 , 36 , 38 , 41 , 51 , 52 ]. In these studies, authors manipulated two or more ad features, including message/information content, tone, amount, or presentation order; images; screen placement; and level of interactivity. Eight of the experimental studies were parallel-group randomized experiments with one group assigned to a manipulated ad exposure in one or more ways and the other group assigned to a control ad exposure [ 7 , 35 , 39 , 40 , 42 , 46 , 47 , 53 ]. In both types of experimental studies, measures of ad engagement varied and included both subjective (e.g., user intentions as to whether they would click the ad or like or share the ad post) and objective measures (e.g., actual click-through rates on ads encountered, view duration tracked by computer). Nearly all studies also measured additional outcomes such as attitudes toward ads, ad or brand recall, or purchase intentions through post-exposure surveys.

3.2 Research question 2: In empirical studies of interactive advertising in naturalistic or real-world contexts, to what extent do consumers engage with interactive advertisements?

3.2.1 study characteristics..

Eight studies addressed RQ 2; these were published between 2006 and 2019 (Table S4-3 in S4 Appendix ) [ 24 , 28 – 31 , 39 , 47 , 54 ]. Six were observational studies [ 24 , 28 – 31 , 54 ], and two studies were experimental but conducted in real-world (i.e., not laboratory) settings [ 39 , 47 ]. The sample sizes across the included studies ranged from 30,638 to 2,000,000 participants. The types of interactive advertisements evaluated varied and could include more than one type of ad. Three studies evaluated banner ads [ 28 , 30 , 47 ], two studies evaluated social network site ads [ 24 , 31 ], and one study evaluated a pop-up ad [ 39 ]. Three studies evaluated other types of digital ads including paid search engine ads and video ads [ 28 , 29 , 54 ]. The type of products advertised across the included studies included unregulated consumer products and services and health/health behavior campaigns.

Authors measured consumer engagement with click-through rates; page views; and/or number of “likes,” comments, or shares on social media. The two experimental studies analyzed click-through rates for banner and pop-up ads [ 39 , 47 ], while the six observational studies analyzed click-through rates for banner ads [ 28 , 30 ], search ads [ 28 – 30 , 54 ], and social media interaction [ 24 , 31 ].

3.2.2 Findings.

An overview of findings is in Fig 3 . The level of engagement by consumers varied across studies. Six studies reported click-through rates ranging from 0.02% to 2.30% [ 24 , 29 , 31 , 39 , 47 , 54 ]. Two of these studies also reported differences in click-through rates when selected characteristics of the ad were varied, such as differences on which page the ad was placed, a variable delay before the ad was displayed [ 39 ], or whether the ads were static or morphing and whether they were context matched to the website on which they were placed [ 47 ]. In contrast to other studies reporting click-through rates, Graham et al. [ 30 ] reported a much higher click-through rate (81.6%); this study used ads to recruit individuals to a website to register for smoking cessation treatment.

https://doi.org/10.1371/journal.pone.0263339.g003

Other measures of consumer engagement beyond click-through rates included number of page views (after clicking through an ad) and interactions such as liking, sharing, or posting comments to ads on social networking platforms. Two studies measured page views, which is the number of pages the viewer visited after going to the landing site [ 29 , 39 ]. In Birnbaum et al. [ 29 ] the median number of pages visited on the website (not including other relevant websites that were linked on the study site) was 1.29. Moe [ 39 ] measured the difference in number of page views when users were exposed to the ad on a gateway page of an informational website compared with exposure to the ad from a content page of the website. The mean number of page views after an ad on a content page (6.31) was higher than page views after an ad on a gateway page (4.86, P < .001), suggesting greater engagement from consumers when involved in the content.

Two studies measured interactive engagement with social media ads through “likes” and shares [ 24 , 31 ]. Horrell et al. [ 24 ] defined levels of consumer engagement as “low” if a consumer liked a page or reacted to a post and “medium” if a consumer shared or commented on a post. Over a 5-week advertising campaign targeted to 91,385 users of a specific Facebook page site targeting lung cancer awareness, the page had 2,602 reactions to posts, 149 page likes, 452 shares, and 157 comments [ 24 ]. Similarly, Platt et al. [ 31 ] reported findings from a 1-month time period in which a Michigan biobank advertising campaign was targeted to an estimated 2 million state residents aged 18 to 28. The campaign’s social media presence garnered 516 page likes, 477 ad likes, 25 page post shares, and 30 entries into an advertised photo contest. This study also reported that a greater percentage of viewers clicked an ad or post they saw when it was associated with the name of a friend who had already liked the Facebook page [ 31 ].

3.3 Research question 3: What is the association between features of interactive advertisements for goods or services and consumer engagement, recall, awareness, or comprehension of product claims and risk disclosures?

3.3.1 study characteristics..

We identified nine studies eligible for RQ 3 that were published between the years 1997 and 2019 (Tables S4-4 and S4-5 in S4 Appendix ) [ 26 , 32 , 34 , 36 , 43 – 45 , 51 , 53 ]. Eight studies were conducted as experiments [ 26 , 32 , 34 , 36 , 43 , 44 , 51 , 53 ], and the remaining study was a meta-analysis [ 45 ]. The sample sizes across the included primary research studies ranged from 60 to 1,600 participants. The type of advertisements evaluated varied. Four studies [ 32 , 34 , 36 , 44 ] evaluated product websites, one study [ 26 ] evaluated banner ads, one study [ 43 ] evaluated both banner ads and advergames, and two studies [ 51 , 53 ] evaluated other types of digital ads (e.g., interactive TV ads and interactive ads in a simulated online store). The included studies manipulated the ad stimuli to vary the level of interactivity or the type of interactive features included in the ad. Interactive features used in these studies included clickable hyperlinks, navigation bars, navigation buttons, rollover and clickable animation, responsive chat features, comment forms, and interactive game elements. The type of products advertised across the included studies included unregulated consumer products and services as well as regulated products or services.

The meta-analysis reported on 63 experimental studies (total N = 13,484) that evaluated how web interactivity affects various psychological outcomes and how those effects are moderated [ 45 ]. Of the included studies, half focused on interactivity within an advertising context, and 25% reported cognition outcomes, the only outcomes of relevance to this review.

3.3.2 Findings.

An overview of findings is in Fig 4 . In the meta-analysis, Yang and Shen [ 45 ] defined cognition measures such as comprehension, elaboration, knowledge acquisition, and recall. The authors reported no significant association between interactivity and cognition (correlation coefficient 0.05, P = .25). Across the eight primary research studies for this RQ, outcomes varied widely by level or type of interactivity. Five of the studies measured consumer recall of the brand, product, or service advertised [ 32 , 34 , 36 , 43 , 44 ]. Four of these involved websites or web pages with varying levels of interactivity [ 32 , 34 , 36 , 44 ].

https://doi.org/10.1371/journal.pone.0263339.g004

In Chung and Zhao [ 36 ], undergraduate university students viewed websites advertising cameras, which were classified as either low, medium, or high interactivity based on the number of hyperlinks included. They found a significant association between a higher number of clicks available and higher memory scores [ 36 ].

In Chung and Ahn [ 32 ], authors asked participants to view either a website with a linear structure (scroll to bottom of page and click link to move to next page), an interactive structure (multiple links available on the page), or a mixed linear and interactive structure and asked them to write down all the product information they could recall after exposure [ 32 ]. The authors found that participants who viewed the linear web page exhibited the highest memory score [ 32 ].

In Macias [ 44 ], participants viewed either a low or high interactivity website that advertised one of two consumer products. The high interactivity websites included rollover animation, hyperlinks, comment forms, and chat features. The authors found that participants who viewed the high interactivity website exhibited greater comprehension [ 44 ].

Polster et al. [ 34 ] reported the results of a study comparing interactive and noninteractive versions of a website with important safety information (ISI) about a fictitious medication viewed either on a desktop computer or smartphone. Authors found that a higher percentage of participants allocated to noninteractive websites saw any ISI as measured through objective clicking and scrolling behavior compared with participants who were allocated to the interactive websites ( P < .001). Further, a higher proportion of desktop-using participants allocated to noninteractive websites recalled at least one relevant side effect compared with participants allocated to the interactive websites ( P < .001) [ 34 ]. A higher proportion of participants using a smartphone allocated to noninteractive websites also had higher recall of at least one relevant side effect compared with participants who were allocated to interactive sites, but this finding was only statistically significant for one of the two noninteractive layouts [ 34 ]. Authors also reported the mean percentage correct recognition of medication side effects and conducted additional analyses of recognition limited to those participants who saw any ISI (Table S4-4 in S4 Appendix ).

Finally, in Daems et al. [ 43 ], Belgian secondary students viewed ads for a fictitious smartphone that were either interactive advergames, static in-game ads, interactive banner ads, or noninteractive banner ads. Authors found that interactive banner ads led to the highest percentage of participants exhibiting brand recognition (60.4%), followed by static in-game ads (22.4%), noninteractive banner ads (21.3%), and finally advergames (14.3%) [ 43 ]. They also found that interactive banner ads led to the highest memory of product characteristics (8.22 out of a 12-point scale), while noninteractive banner ads led to the lowest memory (3.87) [ 43 ].

Three studies measured time spent viewing ads and results were mixed [ 26 , 51 , 53 ]. In Cauberghe and De Pelsmacker [ 51 ], participants from a Belgian market research firm watched a Dutch travel agency interactive TV ad with low, medium, or high interactivity. The interactivity level varied based on the presence of clickable links, navigation bars, and two-way communication. The authors reported significantly more time spent viewing the high interactivity ad (6.1 minutes) than the low interactivity ad (4.4 minutes) [ 51 ]. In Yang [ 53 ], each participant viewed one interactive ad and one noninteractive ad for one of two consumer products. The high interactivity ads offered more user control over order of information, duration of each page, and ability to skip information. Authors found that interactive ads were viewed for less time than noninteractive ads ( P < .01) [ 53 ]. In Muñoz-Leiva et al. [ 26 ], the authors compared “Travel 2.0 websites” with embedded vertical banner ads on 3 different platforms: a Facebook page, a blog, and a Tripadvisor page that varied by level of interactivity [ 26 ]. While the banner ads on all three platforms included a call to action and a clickable link to an airline website, the Facebook ad was the most interactive with the ability to like, comment, and share the ad followed by the blog with the ability to comment on the blog post and finally the Tripadvisor page. The authors used eye-tracking technology to measure the number of visual fixations on the ad, number of seconds until the first fixation on the ad, and total duration of fixations on the ad. They found a significant difference in the number of ad fixations (Facebook, 19.1; blog, 11.7; Tripadvisor (6.1), P < .001). Significant differences were also observed across platforms for other measures (Table S4-5 in S4 Appendix ) [ 26 ].

3.4 Research question 4: How do interactive advertisements for goods and services compare with noninteractive advertisements (e.g., traditional print or broadcast advertisements) with respect to consumer engagement, recall, awareness, and comprehension of product claims and risk disclosures?

3.4.1 study characteristics..

We identified three studies eligible for RQ 4 that were published between the years 2008 and 2018 (Table S4-6 in S4 Appendix ) [ 30 , 48 , 49 ]. One was an observational study [ 30 ], and the other two studies were conducted as experiments. The sample sizes for the two experiments were 233 [ 49 ] and 9,902 [ 48 ] participants; the observational study [ 30 ] did not report the number of persons evaluated. The types of interactive advertisements varied. The observational study [ 30 ] compared banner ads and paid search ads (interactive advertising) with billboards, TV ads, radio ads, outdoor signage, direct mail, and physician detailing (noninteractive advertising). One experimental study [ 48 ] had print flyer, online flyer, and no flyer groups, while the other experimental study [ 49 ] compared a standard TV commercial, a PC advergame, and an interactive TV commercial offering an advergame.

Eligible outcomes for this review reported across the three included studies also varied. The observational study [ 30 ] evaluated outcomes associated with real-world advertising including the number of log-ins and pages viewed, session length, and long-term cookies. Authors of the two experimental studies [ 48 , 49 ] randomized participants to different ad types and evaluated recall in addition to other outcomes such as attitudes, which were not within the scope of this review.

3.4.2 Findings.

An overview of findings is in Fig 5 . Across the three included studies, outcomes varied widely. Graham et al. [ 30 ] examined how online advertising increases consumer demand for smoking cessation treatments in Minnesota and New Jersey (N = NR) by comparing the impact of interactive advertisements (banner ads, paid search ads) versus traditional advertisements (billboards, TV ads, radio ads, outdoor signage, direct mail, physician detailing). Outcomes related to engagement are reported in the RQ 2 section of this review. Ultimately, 9.1% of those who clicked the interactive ad registered for treatment compared with 18.6% of those who were directed to the website from traditional media [ 30 ]. The authors found that compared with traditional ads, online ads engaged a higher percentage of males, young adults, racial/ethnic minorities, individuals with a high school education or less, and dependent smokers. While the authors found significant differences in website engagement metrics (e.g., average session length, pages viewed, percentage posting in public forums) between online and traditional ad responders, they noted that the differences in utilization are too small in magnitude to be meaningful.

https://doi.org/10.1371/journal.pone.0263339.g005

Ieva et al. [ 48 ] estimated the effect of an online versus print promotional advertising flyer on customer response with an experimental design recruiting from a random sample of customers from a supermarket chain (N = 9,902; however, only the 303 that reported viewing the flyer were included in the analysis). The online flyer was a replication of the print flyer with no banners, videos, or embedded links; however, users could click to zoom or move to another page. The authors found no statistically significant differences in recall, recognition, or advertisement memory measures between the online and print flyers.

Bellman et al. [ 49 ] compared the effectiveness of PC advergames, TV commercials, and interactive commercials enhanced with advergames on recall for four test brands of food or personal hygiene products in an experimental study. Members of an Australian audience panel (N = 233) were randomized to one of three ad types. The authors reported significantly higher unaided recall of at least three unique points about the ad content for participants who viewed the PC advergame compared with those who viewed the traditional TV commercial and as compared with the interactive TV ad. Authors observed no significant difference between participants who viewed the interactive TV commercial and the traditional TV commercial.

4. Discussion

4.1 summary of evidence.

Study design and outcomes varied widely within the evidence base for each RQ. That variation itself is noteworthy, as it affects comparability of results and suggests strengths and weaknesses of different approaches for future research in this arena. Through this review, we also can see ways in which existing literature may not yet be optimal for answering questions about consumer risk perception and decision making in response to interactive advertising; much available evidence focuses on indicators of short-term consumer attention in engaging with advertising more than on consumer information processing beyond eyeball movement or click behavior.

Within the 23 studies eligible for RQ 1 (which summarized methods and measures used to evaluate consumer engagement), six were observational studies and 17 were experimental studies. In the experimental studies, methods included within- and between-subjects randomized factorial design and parallel-group randomized experiments. In both types of studies, objective (e.g., click-through rates, eye-tracking metrics) and subjective (e.g., post-campaign surveys, interviews) measures were used to report engagement outcomes. This variability in methods is understandable. Some measures of engagement are most optimally assessed with experimental designs that allow control over content and resource-intensive measurement of respondents (e.g., eye-tracking metrics). Observational studies nonetheless also can offer objective measures of engagement on a larger scale and without the generalizability concerns stemming from volunteer bias inherent to small sample-sized experimental designs. We also did find examples of large-scale experiments [ 39 , 42 ] involving manipulation of advertising stimuli conducted with various kinds of media (digital magazine, websites publishing reviews, news, or information).

Based on this review, consumer engagement is an umbrella concept covering a range of operationalization efforts. The ways in which studies measured engagement reflect 1) varying levels of technologic sophistication of the advertising platform or ad itself, 2) the salience of click-through rates as a metric in commercial advertising (regardless of the theoretical value of that metric to understanding consumer decision making), and 3) varying levels of integration into a broader social media campaign. We did not identify any differences in the way engagement was measured for regulated versus non-regulated products in this scoping review, per se, but the number of studies focused on regulated products or services also was quite limited. Future research on consumer engagement with interactive ads for regulated products should be able to use both observational or experimental designs, depending on the specific outcomes in question.

For RQ 2, eight identified studies reported on the extent to which consumers engage with interactive advertisements in naturalistic or real-world contexts. Consumer engagement was measured with click-through rates; page views; and/or number of “likes,” comments, or shares on social media. Click-through rate was the most common engagement measure used for this RQ; however, the way in which click-through-rates were calculated varied, limiting direct head-to-head comparisons across studies. A click-through rate may offer a conceptually simple way of measuring consumer engagement because it is closely aligned to the evaluation of cost-per-thousand advertisement impressions (i.e., cost-per-mille) and cost-per-click advertising campaigns. In practice, however, variability in click-through rate calculation limits the ability of current literature to offer definitive conclusions related to the concept. Moreover, in the context of evaluating regulated advertising, crude click-through-rates of a single hyperlink in a digital ad may not be enough to provide a nuanced understanding of whether users engage with specific parts of an ad, specifically, claims of benefits, risk disclosures, or both.

For RQ 3, we identified nine studies, eight of which were experiments, that focused on the association between features of interactive ads and consumer engagement, recall, awareness, or comprehension of product claims and risk disclosures. The studies varied the type or level of interactivity in the ad. Some studies found significant associations between higher levels of interactivity and higher memory scores, comprehension, and brand recognition. Other studies found the opposite: better recall and higher memory scores with fewer interactive features. Studies that measured time spent viewing the ads also had mixed results: one study found higher levels of interactivity led to more time spent viewing the ad, whereas one study found the opposite. Further, a meta-analysis reported no correlation between interactivity and measures of cognition.

The evidence for clear relationships between interactive features and outcomes of interest for this scoping review was mixed, precluding any definitive conclusions. Further, some studies addressing this RQ were published during an early era of online advertising that has faded in relevance to present circumstances. Importantly, we also found instances of confounding. In addition to manipulating interactivity, advertisers often manipulated other aspects of the ad not related to interactivity (e.g., tone, text or graphic content). Previous studies have demonstrated that for regulated products, such as prescription drugs, these features moderate consumer understanding of product claims and risk disclosures [ 21 , 56 ]. Thus, future studies evaluating variations in interactive ads of regulated products and services should ensure that study designs and ad manipulations are robust for evaluating independent effects and potential interactions.

For RQ 4, three identified studies compared interactive with noninteractive advertisements with respect to consumer engagement, recall, awareness, and comprehension of product claims and risk disclosures. One observational study found that compared with traditional ads, online ads engaged certain segments of the population better. The two experimental studies found no significant differences between the interactive and traditional ads, but one study found significantly higher unaided recall for participants who viewed a PC advergame compared with those who viewed the traditional or the interactive ads. With the mixed results from this limited number of heterogeneous studies, there is no conclusive evidence on how interactive advertisements compare with noninteractive advertisements with respect to consumer engagement, recall, awareness, and comprehension of product claims and risk disclosures. The limited number of studies may reflect the challenge in conducting direct comparisons of traditional and interactive advertising in the same study. Digital and online advertising offer new and, in some cases, more objective ways of measuring advertising effectiveness that have no counterpart in the evaluation of traditional advertising. Given shifts away from traditional advertising to digital and online marketing because of better returns on investment and ability to target audiences, comparing traditional to interactive ads may not be a relevant comparison for future studies.

4.2 Limitations of evidence

Studies were quite heterogenous with respect to study design, populations evaluated, types of ads used, and measures reported; this limited our ability to conduct a robust synthesis of outcomes. Many studies were conducted among university students; whether findings from such studies would generalize to broader populations is not known. The measures used by some studies to evaluate product or service information recall or knowledge did not appear to be validated. The era over which studies were conducted was broad; some of the interactivity features or platforms used in included studies are likely obsolete or have been replaced by more sophisticated approaches to interactive advertising.

4.3 Limitations of this review

We limited this scoping review to studies published in English from very highly developed countries to increase applicability of findings to policy makers concerned with regulation of interactive advertising in such countries. Study indexing in bibliographic databases was variable and inconsistent; thus, it is possible we missed some relevant studies. Our RQs were focused on outcomes related to consumer engagement with interactive ads, and information recall and comprehension, as it related to product information or risk disclosures. We did not consider consumer attitudes or purchase behavior. We limited measures of engagement to studies conducted in naturalistic or real-world contexts because experimental studies typically manipulated ad exposure or instructed participants what to view and may have put limits on duration of exposure that would not reflect engagement outside of a controlled environment. We did not assess the risk of bias of included studies consistent with a scoping review approach.

4.4 Research gaps

Although the research on interactive advertising is extensive in terms of the volume of available publications, as judged by the size of our initial search yield, the amount of research specifically focused on the influence of interactive advertising on product information recall and specifically risk perception is sparse. Several studies that we screened but excluded as not eligible for this scoping review focused on evaluating tone, content, graphics, placement, or variable deployment of an interactive ad and impact on consumer attitudes about the product or brand or subsequent purchase intention or behavior (see S3 Appendix for a list of excluded studies). Whether such outcomes correlate to an accurate understanding of product features or services and risk disclosures is not known but could be relevant when considering interactive advertising for regulated products, such as prescription drugs, alcohol, tobacco, and financial products or services. Regardless, it is clear that available research on interactive advertising does not provide much of the evidence most useful to regulatory science focused on whether regulated advertising encourages informed decision making about products.

We need rigorously designed studies of consumer experiences with interactive advertising that use objective and validated measures to assess recall and understanding of product or service information and risk disclosures. We note a disjuncture between our selected studies and recent work on social media activities. A type of study we commonly encountered during screening but excluded as not eligible were studies evaluating the impact of influencer marketing through social media. Though not a focus of this scoping review, we noted many of these studies in the latter part of the time period we searched, suggesting an increasing use of this type of digital, interactive advertising for the future and a possible area for future inquiry.

5. Conclusion

This scoping systematic review summarized the research related to consumer engagement with interactive advertisements and impact on recall and understanding of product claims. The evidence shows that consumers do engage with interactive advertisements, but the evidence is mixed as to whether features of interactive advertising increase consumer engagement, recall, awareness, or comprehension of product claims and risk disclosures. Only a few studies compared traditional advertisements with interactive advertisements on these outcomes and these results also were mixed. Some of the limitations of existing interactive advertising literature as a source for informing regulatory science appears to reflect inconsistent labeling of concepts as well as adherence to industry metrics rather than regulatory science needs.

Supporting information

S1 checklist. prisma checklist..

https://doi.org/10.1371/journal.pone.0263339.s001

S1 Appendix. Detailed search strategy.

https://doi.org/10.1371/journal.pone.0263339.s002

S2 Appendix. Study selection criteria.

https://doi.org/10.1371/journal.pone.0263339.s003

S3 Appendix. Excluded studies.

https://doi.org/10.1371/journal.pone.0263339.s004

S4 Appendix. Results tables.

https://doi.org/10.1371/journal.pone.0263339.s005

Acknowledgments

The authors acknowledge Sharon Barrell and Loraine Monroe for editing and document preparation.

• View Article
• Google Scholar
• 5. McMillan SJ. Interactivity is in the eye of the beholder: function, perception, involvement, and attitude toward the web site. In: Shaver MA, editor. American Academy of Advertising; East Lansing, MI: Michigan State University 2000. p. 71–8.
• PubMed/NCBI
• 10. Ha L, James E. Interactivity re-examined: an analysis of business web sites. Conference of the American Academy of Advertising; Washington: Washington State University 1998.
• 11. Rafaeli S. Interactivity: from new media to communication. In: Hawkins R, Pingree S, Weimann J, editors. Advancing communication science: merging mass and interpersonal processes. Newbury Park, CA: Sage; 1988. p. 110–34. pmid:3213354
• 14. Williams F, Rice R, Rogers E. Research methods and the new media. New York: The Free Press; 1988.
• 19. Claiborne AB, Olson S. Strengthening a workforce for innovative regulatory science in therapeutics development: workshop summary: National Academies Press; 2012.

In-app advertising: a systematic literature review and implications for future research

Spanish Journal of Marketing - ESIC

ISSN : 2444-9695

Article publication date: 23 November 2023

The purpose of this study is to review and synthesize the literature on in-app advertising, identify gaps and propose future research directions.

Design/methodology/approach

The authors use a systematic literature review (SLR) approach, following the PRISMA guidelines, to investigate the current state of research in in-app advertising. The study uses 44 shortlisted articles from the Scopus and Web of Science databases. Using the Theory-Context-Characteristics-Methodology (TCCM) framework, the authors analyze the gaps in theory, context, characteristics and methods.

Using thematic analysis, the authors identify five main themes in the in-app advertising literature, namely, ad platform optimization; mobile app user psychology and behavior; ad effectiveness; ad fraud; and security, privacy and other user concerns. The findings show the need for empirical research, with a strong theoretical foundation in emerging ad formats of in-app advertising, user behavior and buy-side of in-app advertising.

Originality/value

This is a maiden study to conduct a domain-based SLR in the emerging field of in-app advertising using the TCCM framework. The authors highlight the key differences between in-app advertising and mobile web advertising. The authors propose theories in the advertising field that could be used in future empirical studies of in-app advertising.

El propósito de esta investigación es revisar y sintetizar la literatura sobre la publicidad en Apps, identificar lagunas y proponer futuras direcciones de investigación.

Utilizamos un enfoque de revisión sistemática de la literatura, siguiendo las directrices PRISMA, para investigar el estado actual de la investigación en publicidad en aplicaciones. El estudio utiliza 44 artículos preseleccionados de las bases de datos Scopus y Web of Science (WoS). Utilizando el marco Teoría-Contexto-Características-Metodología (TCCM), analizamos las lagunas en teoría, contexto, características y métodos.

Conclusiones

Mediante un análisis temático, identificamos cinco temas principales en la literatura sobre publicidad en aplicaciones, a saber: optimización de plataformas publicitarias; psicología y comportamiento de los usuarios de aplicaciones móviles; eficacia publicitaria; fraude publicitario; seguridad, privacidad y otras preocupaciones de los usuarios. Nuestros hallazgos muestran la necesidad de investigación empírica, con una sólida base teórica en los formatos publicitarios emergentes de la publicidad en Apps, el comportamiento del usuario y el buy-side de la publicidad en Apps.

Originalidad

Se trata de un estudio pionero para realizar una revisión sistemática de la literatura basada en el dominio en el campo emergente de la publicidad en Apps utilizando el marco TCCM. Destacamos las principales diferencias entre la publicidad en aplicaciones y la publicidad en la web para móviles. Proponemos teorías en el campo de la publicidad que podrían utilizarse en futuros estudios empíricos sobre la publicidad en Apps.

本研究旨在回顾和总结有关应用内广告的文献, 找出差距并提出未来的研究方向。

我们采用系统性文献综述方法, 遵循 PRISMA 指南, 调查应用内广告的研究现状。研究使用了 Scopus 和 Web of Science (WoS) 数据库中的 44 篇入围文章。利用理论-背景-特征-方法（TCCM）框架, 我们分析了理论、背景、特征和方法方面的差距。

通过主题分析, 我们确定了应用内广告文献的五大主题, 即广告平台优化; 移动应用用户心理和行为; 广告效果; 广告欺诈; 安全、隐私和其他用户关注点。我们的研究结果表明, 有必要在应用内广告的新兴广告形式、用户行为和应用内广告买方等方面开展实证研究, 并奠定坚实的理论基础。

这是一项首次使用 TCCM 框架对新兴的应用内广告领域进行基于领域的系统性文献综述的研究。我们强调了应用内广告与移动网络广告的主要区别。我们提出了广告领域的理论, 可用于未来的应用内广告实证研究。

• In-app advertising
• Mobile app advertising
• Mobile marketing
• Systematic literature review
• Publicidad en aplicaciones
• Publicidad en aplicaciones móviles
• Marketing móvil
• Revisión sistemática de la literatura
• - 应用内广告 移动应用广告 移动营销 系统文献综述

Maddodi, C.B. and Upadhyaya, P. (2023), "In-app advertising: a systematic literature review and implications for future research", Spanish Journal of Marketing - ESIC , Vol. ahead-of-print No. ahead-of-print. https://doi.org/10.1108/SJME-05-2022-0120

Emerald Publishing Limited

Copyright © 2023, Chetana Balakrishna Maddodi and Pallavi Upadhyaya.

Published in Spanish Journal of Marketing - ESIC. Published by Emerald Publishing Limited. This article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial and non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode

1. Introduction

The exponential growth and penetration of mobile (6.5 billion smartphone users globally as per Statista, 2022 ) have bestowed unique opportunities to use it as an advertising medium. Lately, the focus of mobile marketing has moved from the mobile web to mobile apps ( Keem and Lee, 2018 ), and in-app advertising has emerged as a new conduit for mobile advertising. The Interactive Advertising Bureau (2022) defines in-app advertising as “ads and ad campaigns that are delivered within mobile applications, including smartphones, tablets, or wearable devices”. Sensor Tower has estimated that global in-app advertising spending will reach US$233bn by 2026 ( Chan, 2022 ). Mobile apps are popular, as 97% of the apps are free to download ( Statista, 2023 ), easily accessible round the clock and gratify a range of specific user needs ( Sigurdsson et al. , 2018 ). In-app advertisements have become an important source of revenue for app developers ( Gao et al. , 2022 ; Mattke et al. , 2021 ; Truong et al. , 2019 ).

However, researchers posit that there is little understanding of the differences between the two mobile channels: mobile web and mobile apps ( Park and Park, 2020 ). Mobile apps are considered catalysts for “new” customer experiences and engagement, delivering a unique source of customer value. The “always on” points of interaction in mobile apps shape positive and interactive customer journeys ( Stocchi et al. , 2022 ). Mobile apps are developed for a particular operating system, whereas mobile websites are delivered through smartphone browsers. According to e-marketer, the in-app advertisement viewability rate exceeds mobile web and desktop viewability worldwide ( Handel, 2022 ). The key differences between mobile web and in-app advertisements are summarized in Appendix 1 . Compared to mobile web advertisements, in-app advertisements have a captive and more engaged audience. Targeted and personalized advertisements, with higher accuracy, can be placed due to the availability of location, demographics and app content.

Mobile advertising exists in many forms, such as web-based, mobile search, in-app, cross-app, location-based and social media ( Jebarajakirthy et al. , 2021 ). Maseeh et al. (2021) , in their recent meta-analysis of mobile advertising research, avow the underrepresentation of empirical studies on in-app advertising in the literature. Although in-app advertising has been celebrated for revolutionizing the advertising landscape, scholarly attention on in-app advertising is limited when compared to other domains of mobile advertising ( Truong et al. , 2019 ).

Although there are a few comprehensive systematic literature reviews (SLRs) on mobile marketing, in general ( Leppäniemi et al. , 2006 ; Narang and Shankar, 2019 ; Varnali and Toker, 2010 ) and specifically on mobile advertising ( Billore and Sadh, 2015 ; Grewal et al. , 2016 ; Jebarajakirthy et al. , 2021 ; Maseeh et al. , 2021 ), there is limited discussion on issues in in-app advertising. Previous reviews in the domain have focused on the broader perspective of mobile advertising and a summary of literature reviews in the mobile advertising domain is presented in Appendix 2 . Truong et al. ’s (2019) review of in-app advertising specifically examines the relationships between factors controlled by publishers and the effectiveness of in-app advertising. However, there are no SLRs that examine the broader research issues in in-app advertising research.

What are the key overarching themes latent in the literature on in-app advertising?

What are the most prevalent theories, contexts, characteristics and methods used in the in-app advertising literature?

What are the future areas of research on in-app advertising?

This review makes several contributions to the emerging research area of in-app advertising. First, the review is the maiden attempt to examine the extant literature on in-app advertising and identifies five key themes. Second, the study identifies research gaps in the theories, characteristics, context and methods used in the in-app advertising literature. Third, the study proposes future areas of research and identifies potential theories that could shed light on new dimensions of in-app advertising.

2. Review methodology

The study uses a SLR, as suggested by Transfield et al. (2003) . Systematic studies seek to discover, examine and summarize the contents in an integrative approach, thereby making the evidence more conducive for scholarly audit ( Bergman and Holden, 2010 ; Transfield et al. , 2003 ). SLRs are suitable for niche research areas with a narrow scope ( Donthu et al. , 2021 ) and have a smaller number of papers for review ( Snyder, 2019 ). Donthu et al. (2021) posit that SLR methodology is often used when the review goal is to summarize and synthesize findings of a research topic and the volume of the data set for review is small for manual review.

Paul and Criado (2020) broadly categorize SLRs into domain-based, theory-based and method-based reviews. The authors posit that a domain-based review can be termed a framework-based review if the study adopts an existing framework or develops a framework and uses it to organize the review. In this study, we perform a framework-based SLR with 44 shortlisted articles on in-app advertising. We adopt the Theory-Context-Characteristics-Methodology (TCCM) framework developed by Paul and Rosado-Serrano (2019) for examining and analyzing the in-app advertising literature and presenting the findings, gaps and future areas of research. The TCCM framework is extensively used in marketing to examine the literature and present the findings of the review ( Bhattacharjee et al. , 2022 ; Celik et al. , 2022 ; Jebarajakirthy et al. , 2021 ; Singh and Dhir, 2019 ). In this paper, we use this framework to present theories on in-app advertising (used in the literature and potential for future research), context (country, industry, advertising format), characteristics (antecedents and outcomes, mediators and moderators) and methods (research approaches and analysis techniques).

We perform thematic analysis on the selected articles to identify themes in the in-app advertising literature. Thematic analysis is a method for identifying, analyzing and recording patterns within the data ( Braun and Clarke, 2006 ) and synthesizing them into themes ( Thomas and Harden, 2008 ). We followed the process outlined by Braun and Clarke (2006) and Thomas and Harden (2008) to conduct a thematic analysis as presented in Section 3.

2.1 Search strategy, information retrieval and information processing

The PRISMA guidelines have assisted the article search and selection process, as followed by Page et al. (2021) . To address the aforementioned research questions, we follow a three-step process to identify the relevant studies. The first step is to determine an appropriate database to retrieve documents for the study. Articles were sourced from the Scopus and Web of Science databases, and articles until the end of May 2023 were included in this review. The two journal repositories are considered the largest and most widely used online databases and search engines by researchers ( Paul et al. , 2021 ). Peer-reviewed journals in the English language constitute the prime source of articles included in this study.

The second step consists of selecting appropriate keywords to extract relevant documents. The search keywords “in-app advertisements”, “in-app ads”, “in-app advertising”, “mobile app advertisements”, “mobile app advertising”, “mobile app ads” and “mobile application advertising” were selected to exemplify the line of inquiry effectively. In the third step, the data set was gathered and refined for the study.

The search method of title, abstract and keywords was used with Boolean operators of relevant keywords. One additional round of search iteration with the cited references was carried out to ensure the inclusion of relevant research studies in the area. The PRISMA protocol for reporting literature in systematic reviews was used to report the search strategy ( Page et al. , 2021 ). Figure 1 outlines the methodology of the SLR. The initial search yielded 209 studies published in the two databases.

The data retrieved were exported to Microsoft Excel for scrutiny and 105 studies were shortlisted for SLR after removing duplicates and non-English studies from the two databases. Book chapters, conference papers, editorials and lecture notes were excluded because they were not necessarily peer-reviewed and the exclusion resulted in 56 records. To winnow down the out-of-scope studies, abstract screening was performed and 12 records were excluded because they did not encapsulate the essence and conceptual boundary of in-app advertising. Thus, 44 studies were included in the final analysis of this review.

3. Thematic analysis of in-app advertising literature

Research on in-app advertising spans several areas, such as information technology, consumer behavior and psychology, marketing, advertising and communication. We performed a thematic analysis of the shortlisted articles to identify the broad areas of research in the in-app advertising literature. The coding of text and generating and reviewing themes were performed based on the thematic analysis process suggested by Braun and Clarke (2006) . Each shortlisted article was first coded manually by examining the title and abstract by the first author. Seventeen descriptive themes emerged by grouping similar codes. Figure 2 illustrates the thematic map emerging from the literature. The descriptive themes were labeled based on their significance and context in relation to the findings of the primary studies. The descriptive themes were then verified for relevance by the second author and finalized after discussion.

In the next phase, the analytical themes were inferred from the descriptive themes ( Thomas and Harden, 2008 ) and five themes emerged in the discussions between the authors: ad platform optimization; mobile app user psychology and behavior; ad effectiveness; ad fraud; and security, privacy and other user concerns.

The thematic analysis indicates the relationship between the two themes in the literature, namely, ad platform optimization – ad effectiveness ( Truong, 2023 ) and ad effectiveness – mobile app user behavior ( Mattke et al. , 2021 ). The click-through rate was one of the outcome variables that was investigated by several researchers for the optimal combination of in-app advertisement factors and the optimal sequencing of in-app advertisements ( Adikari and Dutta, 2020 ; Sun et al. , 2017 ; Truong et al. , 2019 ). The relationship between the themes has explored the factors controlled by advertisers, consumers and ad networks on click behavior. However, the relationship between other themes was not explored in past literature.

3.1 Theme 1: ad platform optimization

The in-app advertising ecosystem is divided into players on the buy-side or demand-side and the sell-side or supply-side. Optimizing the in-app advertising platform is of utmost importance, keeping in view the interest of both buy-side and sell-side players. Each player in the in-app advertising ecosystem has varied outcomes of interest ( Truong et al. , 2019 ).

Publishers can either sell their inventory directly to ad networks or buy-side or engage in real-time bidding ( Kaplan, 2021 ). Truong (2023) found that the optimal combination of publisher-controlled factors such as duration, timing, size and position of the ad space significantly improves the click-through rate of in-app advertising.

Several recent studies have examined the techniques and guidelines for app developers to assist them balance the trade-offs in-app advertising costs and benefits. Ghose and Han (2014) find that demand for mobile apps decreases with in-app advertisements and suggest several app design choices for app developers. App developers often integrate ad libraries with apps and there are more than 63 ad libraries ( Ahasanuzzaman et al. , 2020 ) in the Google Play Store. Ahasanuzzaman et al. (2022) develop a reference architecture for ad library integration for app developers.

The ad networks connect publishers and advertisers, allowing advertisers to bid on the ad space provided by the publisher. Although publishers may have many ad networks ready to buy ad inventories, the performance of networks differs. Adikari and Dutta (2020) have suggested an optimal mechanism for publishers to select the best ad network. Recent research ( Lee et al. , 2021 ; Rafieian and Yoganarasimhan, 2020 ; Sun et al. , 2017 ; Yang and Cao, 2018 ) examines efficient techniques for ad networks using machine learning techniques.

Ad exchange orchestrates the buying and selling of ads and can connect publishers with multiple ad networks. Revenue optimization is a vital task for an ad exchange. Scholars have examined techniques to improve ad auction mechanisms ( Liu and Liu, 2019 ) and optimal revenue-sharing methods ( Hao et al. , 2016 ; Hu et al. , 2023 ). Mukherjee et al. (2017) develop a machine learning-based in-app ad selection algorithm and propose an optimization model to maximize supply-side platform revenue.

The current research on this theme is primarily exploratory and uses data-driven approaches. There is no theoretical framework applied to the studies and machine learning techniques are prominent. Researchers provide either a novel optimal solution or suggest an extension of an existing technique. Although the sell-side platform has gained traction, academic literature on the buy-side is sparse. An exception is the field research by Rosenkrans and Myers (2018) that proposes how predictive analytics could be used by advertisers to optimize mobile location-based ads. The optimization of advertisements based on ad characteristics such as design, context, size and media types and their influence on information processing and consumer response remain unexplored.

3.2 Theme 2: mobile app user psychology and behavior

This theme primarily focuses on studies on attitude, behavioral intention and user behavior toward in-app advertising. Attitudes toward in-app advertising have been examined by several researchers and found to impact people’s intention to use the app ( Sigurdsson et al. , 2018 ). Logan (2017) posits that attitude varies according to the gratification sought from the mobile app category. Scholars posit that the perceived value of in-app advertising, trust toward ads, ad source and perceived irritation toward ads are significant factors affecting attitudes toward advertising ( Aydin and Karamehmet, 2017 ). Users’ propensity to trust affects their trust in in-app advertising and trust impacts attitudes toward in-app ads and their intention to view in-app ads ( Cheung and To, 2017 ). Attitude also mediates the relationship between trust and purchase intentions ( Sung, 2020 ). Ad factors such as credibility, information and perceived entertainment considerably improve attitudes toward in-app ads ( Sigurdsson et al. , 2018 ).

The behavioral response toward ads is better when there is congruity between the ad and the mobile environment ( Wang and Chou, 2019 ). Mattke et al. (2021) conduct a qualitative comparative analysis and empirically show the sequential processing of structural and semantic factors of in-app advertisements. Keem and Lee (2018) examine the types of advertisements consumers prefer or adversely react to when shopping through mobile apps.

Consumers control their in-app advertising performance through their behavior and personal characteristics ( Truong et al. , 2019 ). However, studies on personal characteristics and their impact on in-app advertising are scarce. User psychology and behavior can vary based on the type of app used, user loyalty and ad formats. In addition, there is ample opportunity to understand aspects of user psychology and behavior through theoretical lenses.

3.3 Theme 3: ad effectiveness

Rodgers and Thorson (2000) categorize all factors affecting advertising effectiveness into factors controlled by consumers and advertisers. The researchers posit that attention, memory, attitude and outcomes are controlled by consumers. Therefore, these are important for advertisers to measure ad effectiveness. Ad recall is one of the popular metrics used to measure ad effectiveness. Cicek et al. (2018) examine the effects of banner location, app type and orientation on ad recall and find that banner ads positioned at the top can trigger better recall.

Scholars have examined advertiser-controlled ad characteristics, ad content characteristics ( Cassioli, 2019 ; Sigurdsson et al. , 2018 ; Wang and Chou, 2019 ), interactivity ( Cassioli and Balconi, 2022 ) and media type ( Sung, 2021 ; Sung et al. , 2022 ). Several studies have examined the impact of ad characteristics on outcome metrics such as monetization ( Appel et al. , 2020 ; Ji et al. , 2019 ; Nuwamanya et al. , 2018 ; Rutz et al. , 2019 ) and user engagement ( Cassioli, 2019 ; Cassioli and Balconi, 2022 ).

The literature addresses only a few consumer/advertiser factors, as noted above. The impact of creativity on ad effectiveness has received less attention. Moreover, publisher-controlled factors (ad timing, ad position) and network-controlled factors (location, device type and size, which impact ad effectiveness) remain unexplored.

3.4 Theme 4: ad fraud

The rise in popularity of free and open-source Android platforms for mobile has resulted in fraudulent and repackaged apps. Repackaged apps are modified versions of existing popular apps, which are designed to capture advertising revenue and obtain user information. The lack of in-app ad-blocking solutions among mobile users has led to abusive in-app advertising practices such as click fraud and malvertising. App developers can generate fraudulent ad clicks in an automated manner, resulting in losses for advertisers. Recent studies have examined the vulnerability of ad networks to click fraud attacks and proposed techniques for ad networks to detect click fraud ( Cho et al. , 2016 ; Mouawi et al. , 2019 ).

Cho et al. (2016) note that there is a dearth of research on the vulnerability of ad networks to different varieties of ad click threats and methods to prevent them. New forms of ad fraud, such as ad stacking, click flooding, device ID reset fraud, bot fraud, app fraud, bundle ID spoofing and software development kit (SDK) spoofing ( TrendMicro, 2019 ), have received little attention from researchers. There are ample research opportunities to leverage developments in machine learning algorithms and artificial intelligence to combat these threats.

3.5 Theme 5: security, privacy and other user concerns

Studies have examined the security, privacy and smartphone performance concerns of mobile users resulting from in-app advertising. A mobile app embeds a custom SDK given by an ad network to display ads. This gives greater control over the ad network on the mobile application. Scholars ( Su et al. , 2018 ) observe that in-app advertising poses a higher security risk to users by collecting their personal information, downloading malware in the background, disclosing images and personal data and exposing them to inappropriate content. Researchers have voiced their concerns about the noncompliance of app developers with existing relevant international guidelines or technology standards ( Haenschen and Wolf, 2019 ; Scott et al. , 2018 ) and the need for regulation of ad libraries. Several studies have proposed techniques to detect malvertising in ad networks ( Shao et al. , 2018 ) and prevent third-party libraries from misusing the permissions of host apps ( Hsu et al. , 2020 ). Recent research ( Gao et al. , 2021 , 2022 ) has identified user concerns with in-app advertising using app reviews.

Our review signals a lack of research on the brand safety of advertised brands and brands powering the app. Further research is needed on improper content, dark patterns in apps and their impact on the brand. Behavioral ad avoidance, such as closing or uninstalling the mobile app, due to content inappropriateness of in-app advertising and dark patterns, is a major concern to publishers as well as advertisers. Future research can explore the impact of such issues on attitudes toward the ad and brand powering the app. Threats posed by in-app advertising among vulnerable groups such as children are another unexplored area.

4. Theory-Context-Characteristics-Methodology in in-app advertising: findings and discussion

In accordance with the TCCM framework developed and applied by Paul and Rosado-Serrano (2019) and following the examples of other TCCM reviews ( Billore and Anisimova, 2021 ; Celik et al. , 2022 ; Ghorbani et al. , 2022 ; Jebarajakirthy et al. , 2021 ), we present a comprehensive outline of the findings and gaps existing in the literature.

In the theoretical perspective section, we present the most commonly used theoretical frameworks in the in-app advertising literature. In the context section, we present the research settings (country, industry) used in the in-app advertising literature and the diverse types of ad formats used. In the characteristics section, we examine the antecedents, outcomes, mediators and moderators used in in-app advertising studies. We summarize the TCCM framework for in-app advertising in Figure 3 .

4.1 Theoretical perspectives in the in-app advertising literature (T)

According to our review, previous studies in in-app advertising have not used theoretical frameworks to understand the unique complexities and features of in-app advertising. Only 11 studies (approximately 25% of the studies) in the in-app advertising literature have used theoretical underpinnings. The studies are primarily on the themes of “mobile app user psychology and behavior” and “ad effectiveness”. Table 1 presents the theories applied in the in-app advertising literature.

Theoretical integration of multiple paradigms could serve to better understand and interpret the complex nature of the advertising phenomenon in digital media ( Rodgers and Thorson, 2000 ). For effective advertising research, both exploratory research and theory-based research are important ( Huh, 2017 ); hence, future researchers should establish connections between existing advertising theories and computational advertising research.

The review highlights that in-app advertising has relied on nine theories, as described in Table 2 . Uses and gratification theory was the most commonly used theory in the in-app advertising literature, with four studies applying it in their research. There is potential to use several advertising theories to examine new dimensions of in-app advertising research. We propose a few of them in Section 5.

4.2 Context of the studies in in-app advertising (C)

The TCCM framework regard “context” as circumstances forming the research setting ( Paul and Rosado-Serrano, 2019 ). We analyzed the countries, industries and ad formats that have appeared in the in-app advertising literature. In-app advertising has been researched in 16 different countries. Data were collected from US participants in 30% of studies, followed by China, (11%), Canada, India, Taiwan (approximately 7% each, i.e. 20%), Hong Kong, Italy, Korea, Turkey, UK, Vietnam (approximately 5% each, i.e. 28%) and Australia, Germany, Singapore, Lebanon, Malaysia, Uganda (approximately 2% each, i.e. 11%). Most of the studies used data from participants from a single country, except for studies by Sigurdsson et al. (2018) , which used participants from two countries.

In the context of industries, in-app advertising research has been explored in tourism ( Sung, 2020 ), health care ( Nuwamanya et al. , 2018 ), education ( Hsu et al. , 2020 ), news media ( Cassioli, 2019 ) and gaming ( Jiao et al. , 2022 ; Rutz et al. , 2019 ; Wang and Chou, 2019 ). However, industries such as financial services, retailing, hospitality, media and entertainment have yet to be explored. The financial services industry extensively uses in-app advertising to upsell and cross-sell financial products and services ( Upshot.ai, 2022 ). Along similar lines, hospitality, retailing, media and entertainment use in-app advertising for both marketing and revenue generation. Hence, there is a vast scope for research in these industries.

In the context of advertisement formats, studies have investigated contextual advertising ( Graham et al. , 2021 ), banner advertising ( Cicek et al. , 2018 ), interstitial advertising ( Wang and Chou, 2019 ), short messaging services ( Aydin and Karamehmet, 2017 ) and augmented reality advertising ( Sung, 2021 ; Sung et al. , 2022 ). However, the Interactive Advertising Bureau (2022) lists many more advertisement formats, such as reward ads, native ads, video ads, emoji ads, virtual reality ads and audio ads. Future research can explore these emerging advertisement formats.

4.3 Characteristics of in-app advertising studies (C)

We examine the empirical research on in-app advertising that used theoretical frameworks and synthesized the antecedents, outcomes, mediators and moderators of in-app advertising in this section. Studies reveal that advertisement-related antecedents such as advertising content, animation and interactivity influence advertising value and have significant effects on emotional engagement ( Cassioli, 2019 ) and visual behavior ( Cassioli and Balconi, 2022 ). Antecedents such as informativeness, credibility, entertainment ( Aydin and Karamehmet, 2017 ), personalization ( Sigurdsson et al. , 2018 ), trust ( Cheung and To, 2017 ), attitudinal beliefs ( Hsu et al. , 2020 ) and gratification sought ( Logan, 2017 ) influence attitudes toward in-app advertisements. Attitude influences the intention to watch in-app advertisements ( Cheung and To, 2017 ), the intention to adopt m-learning platforms with in-app advertisements ( Hsu et al. , 2020 ) and the behavioral intention to click on in-app advertisements ( Sigurdsson et al. , 2018 ), which in turn influences user behavioral responses. Self-efficacy, the perceived value of collaboration and incentives influence the intention to interact with in-app ads ( Qureshi et al. , 2022 ). Cicek et al. (2018) provides causal evidence that app type, app orientation and banner ad location influence ad recall. Mattke et al. (2021) studied structural factors of in-app advertising, including animation, color brightness, location prominence, size and similarity to background structure, and semantic factors, including perceived entertainment value, informativeness, irritation, privacy concern, credibility, congruity and personalization, to investigate the consequences of clicking behavior. The influence of antecedents such as advertisement types affects the click-through rate ( Rosenkrans and Myers, 2018 ), advertising avoidance behaviors ( Keem and Lee, 2018 ), purchase intention ( Sung, 2021 ) and social media sharing ( Sung et al. ,2022 ). The interactive effects of publisher-controlled factors, namely, ad space size, ad space position ad space timing ( Truong et al. , 2019 ), and ad space duration ( Truong, 2023 ), with advertiser-controlled factors, namely, ad type, ad medium, location ( Truong et al. , 2019 ) and time ( Truong, 2023 ), on the click-through rate were investigated. Wang and Chou (2019) examine how the degree of congruity between promoted products in interstitial ads and the app environment affects consumer response toward the ad.

Mediator constructs such as trust, intention to watch in-app advertisements ( Cheung and To, 2017 ), attitude ( Sigurdsson et al. , 2018 ), ad response ( Sung, 2020 ), perceived value of collaboration and incentives ( Qureshi et al. , 2022 ) were used to study the impact on behavioral intention. Wang and Chou (2019) investigated the moderating effects of game types and game immersion on advertising effects. Aydin and Karamehmet (2017) examined the mediating effect of advertising value on attitude. Qureshi et al. (2022) assessed the moderating effect of self-efficacy on the intention to interact with in-app advertisements. Based on the synthesis of the literature, we found that with reference to outcomes, clicking behavior, behavioral intention and attitude are the most frequently studied consequences.

In the current studies, antecedents related to consumer-related factors such as personality traits, memory, feelings of intrusiveness, psychological reactance and their impact on behavioral outcomes are scarce. Understanding the unintended consequences of in-app advertising is important, and future studies can focus on reducing negative consumer reactance using an optimal combination of publisher-controlled, consumer-controlled and advertiser-controlled ( Mattke et al. , 2021 ) antecedents. In future research, there is also a need to investigate the moderating effect of demographic variables such as age, income and gender on clicking behavior. Empirical studies using theoretical frameworks to examine user behavior in in-app advertising are scarce.

4.4 Methods in the in-app advertising literature (M)

The methodology-wise analysis suggests that quantitative research methods were prominent (93 % of the total). Only two studies ( Haenschen and Wolf, 2019 ; Scott et al. , 2018 ) used a qualitative approach, and one study ( Logan, 2017 ) used a mixed method.

As in-app advertising is an interdisciplinary topic, 50% of the studies were from the field of information technology. The methods in these studies were simulation experiments and the development of effective algorithms, optimization models, econometric models or machine learning models. The analysis techniques used in these studies examined the effectiveness of these models, and in two studies ( Gao et al. , 2021 , 2022 ), text mining was used.

The remaining 50% of the total studies were from marketing, advertising and consumer behavior. The methods in these studies include surveys (9), laboratory and field experiments (7), neuroscientific experiments (2), qualitative (2) and mixed methods (1). The analysis techniques used in these studies include SEM, ANOVA and descriptive analysis.

As in-app advertising is still emerging, a mixed-method approach can provide a comprehensive understanding of an emerging phenomenon.

5. Future research agenda

There are several gaps in the extant literature that provide avenues for future research. From a theoretical perspective, we observe that research in in-app advertising has not been examined through several theoretical lenses. Future empirical research with theoretical foundations would advance the understanding of consumer behavior in this emerging area. Rodgers and Thorson (2019) offer a comprehensive synthesis of the vast literature on advertising and provide theoretical approaches to advertising. Considering the unique characteristics of mobile apps and the high level of user engagement compared to other mediums, there is scope for applying several advertising theories. We propose five advertising theories in Table 2 for future research on in-app advertising.

There is also ample scope for examining underexplored contexts (country, industry and ad format) in in-app advertising. As research from the USA dominates the literature, future research may examine app user behavior from Asia-Pacific and European regions. Country-specific issues examining the regulation and compliance of app developers could be undertaken. As there are only a few studies that examine the buy-side or publishers of in-app ads, future research may examine publisher-controlled elements and their impact on ad effectiveness and user behavior. As there are novel formats of in-app ads such as virtual reality ads and many multimedia ad formats, future research may examine issues such as ad avoidance, ad fraud and ad effectiveness in emerging ad types in in-app advertising.

Only a few users’ behavior constructs in the in-app advertising context have been studied in the literature. Future studies may examine user demographics, personality traits, motivation, attention and their impact on attitude and ad avoidance. Age and gender can be examined as moderating variables, as they can also influence attitudes toward in-app advertising.

The mobile app is a pull medium and users are highly goal-oriented compared to other mediums. Hence, consumer-controlled aspects ( Rodgers and Thorson, 2000 ) of mobile apps, such as motivation, attention, involvement and their impact on cognitive and emotional processing of in-app advertising, may be explored in future studies.

From a methodological perspective, analytical and quantitative research have dominated the literature. Future empirical research with a theoretical foundation and mixed methods research approaches (both quantitative and qualitative) could be used. Case studies in in-app advertising could also provide more insights into the in-app advertising phenomenon. New machine learning and artificial intelligence techniques need to be researched to combat ad fraud. Table 3 highlights the key research gaps in theories, context, characteristics and methodology in the in-app advertising literature and proposes future research directions.

6. Theoretical and practical implications and conclusions

This study makes three main theoretical contributions to the growing literature on mobile advertising and digital marketing. First, the study is a maiden attempt to systematically review the literature on in-app advertising and synthesize it into five broad themes. The thematic analysis reveals that there are several research gaps in the in-app advertising literature and there is a need for empirical research on emerging ad formats of in-app advertising, user behavior and issues on the buy-side of in-app advertising. As in-app advertising is a fast-growing mobile advertising format, understanding consumer behavior and its effectiveness in different contexts is essential. We highlight some of the key differences between in-app advertising and mobile web advertising to spark interest in future researchers and undertake further studies.

Second, the study identifies the gaps in the literature and proposes future research areas (refer to Table 3 ) in in-app advertising based on the TCCM framework. The framework (refer to Figure 3 ) developed for in-app advertising gives a holistic view of the body of knowledge. It presents the antecedents, outcomes, moderators and mediators in the current literature.

Third, we propose five advertising theories (refer to Table 2 ) that could guide future researchers to investigate issues in in-app advertising. We also highlight future research directions using the proposed theories.

The study has practical implications for advertisers and other stakeholders in the in-app advertising ecosystem. The review provides practitioners with an overview of the in-app advertising research that could be used to develop more effective strategies for its deployment. The review presents practitioners with an unbiased summary of in-app advertising research that can guide advertisers in designing and delivering in-app advertisements. By providing insights into the privacy and security concerns of users, we encourage practitioners to be mindful of using ethical practices in in-app advertising.

One of the limitations of the current review is the paucity of empirical studies. Quantitative analyses such as bibliometric analysis and meta-analysis could not be performed due to the heterogeneity of constructs and the paucity of empirical studies ( Donthu et al. , 2021 ). Future research may examine a single phenomenon/construct in in-app advertising.

In-app advertising is set to be a key driver of mobile advertising over the coming years. Advances in mobile technology coupled with industry challenges such as ad fraud, ad avoidance and ad effectiveness measurement provide many opportunities for future research in the in-app advertising domain.

Flow diagram of systematic literature review on in-app advertising

Thematic map of in-app advertising literature

TCCM framework for in-app advertising

Theoretical perspectives used in the in-app advertising literature

Proposed advertising theories for future research in in-app advertising

Future directions for research in in-app advertising

In-app versus mobile web advertising

Summary of literature reviews in mobile advertising in the past decade

Adikari , S. and Dutta , K. ( 2020 ), “ Adaptive ad network selection for publisher-return optimization in mobile-app advertising ”, Decision Sciences , Vol. 52 No. 4 , pp. 986 - 1017 .

Ahasanuzzaman , M. , Hassan , S. and Hassan , A.E. ( 2022 ), “ Studying ad library integration strategies of top free-to-download apps ”, IEEE Transactions on Software Engineering , Vol. 48 No. 1 , pp. 209 - 224 .

Ahasanuzzaman , M. , Hassan , S. , Bezemer , C.P. and Hassan , A.E. ( 2020 ), “ A longitudinal study of popular ad libraries in the google play store ”, Empirical Software Engineering , Vol. 25 No. 1 , pp. 824 - 858 .

Ajzen , I. ( 1991 ), “ The theory of planned behavior ”, Organizational Behavior and Human Decision Processes , Vol. 50 No. 2 , pp. 179 - 211 .

Appel , G. , Libai , B. , Muller , E. and Shachar , R. ( 2020 ), “ On the monetization of mobile apps ”, International Journal of Research in Marketing , Vol. 37 No. 1 , pp. 93 - 107 .

Aydin , G. and Karamehmet , B. ( 2017 ), “ A comparative study on attitudes towards SMS advertising and mobile application advertising ”, International Journal of Mobile Communications , Vol. 15 No. 5 , pp. 514 - 536 .

Bauer , C. and Strauss , C. ( 2016 ), “ Location-based advertising on mobile devices: a literature review and analysis ”, Management Review Quarterly , Vol. 66 No. 3 , pp. 493 - 509 .

Bergman , E.M. and Holden , I.I. ( 2010 ), “ User satisfaction with electronic reference: a systematic review ”, Reference Services Review , Vol. 38 No. 3 , pp. 493 - 509 .

Bhattacharjee , D.R. , Pradhan , D. and Swani , K. ( 2022 ), “ Brand communities: a literature review and future research agendas using TCCM approach ”, International Journal of Consumer Studies , Vol. 46 No. 1 , pp. 3 - 28 .

Billore , S. and Anisimova , T. ( 2021 ), “ Panic buying research: a systematic literature review and future research agenda ”, International Journal of Consumer Studies , Vol. 45 No. 4 , pp. 777 - 804 .

Billore , A. and Sadh , A. ( 2015 ), “ Mobile advertising: a review of the literature ”, The Marketing Review , Vol. 15 No. 2 , pp. 161 - 183 .

Braun , V. and Clarke , V. ( 2006 ), “ Using thematic analysis in psychology ”, Qualitative Research in Psychology , Vol. 3 No. 2 , pp. 77 - 101 .

Brehm , J.W. ( 1966 ), A Theory of Psychological Reactance , Academic Press, American Psychological Association , New York, NY .

Calder , B.J. and Malthouse , E.C. ( 2009 ), “ Media engagement ”, in Gröppel-Klein , A. and Germelmann , C.C. (eds), Medien im Marketing , Gabler .

Cassioli , F. ( 2019 ), “ Contents, animation, or interactivity: neurophysiological correlates in app advertising ”, Neuropsychological Trends , Vol. 26 No. 26 , pp. 83 - 91 .

Cassioli , F. and Balconi , M. ( 2022 ), “ Advertising in app: a neuroscientific approach ”, International Journal of Internet Marketing and Advertising , Vol. 16 No. 3 , pp. 257 - 270 .

Celik , F. , Kam , S.M. and Köseoglu , M.A. ( 2022 ), “ Ad avoidance in the digital context: a systematic literature review and research agenda ”, International Journal of Consumer Studies , pp. 1 - 35 .

Chan , S. ( 2022 ), “ 5-Year market forecast ”, available at: www.sensortower.com/blog/sensor-tower-app-market-forecast-2026 (accessed 20 February 2023 ).

Cheung , M.F. and To , W. ( 2017 ), “ The influence of the propensity to trust on mobile users' attitudes toward in-app advertisements: an extension of the theory of planned behavior ”, Computers in Human Behavior , Vol. 76 , pp. 102 - 111 .

Cho , G. , Cho , J. , Song , Y. , Choi , D. and Kim , H. ( 2016 ), “ Combating online fraud attacks in mobile-based advertising ”, EURASIP Journal on Information Security , Vol. 2016 No. 1 , pp. 1 - 9 .

Cicek , M. , Eren-Erdogmus , I. and Dastan , I. ( 2018 ), “ How to increase the awareness of in-app mobile banner ads: exploring the roles of banner location, application type and orientation ”, International Journal of Mobile Communications , Vol. 16 No. 2 , pp. 153 - 166 .

Donthu , N. , Kumar , S. , Mukherjee , D. , Pandey , N. and Lim , W.M. ( 2021 ), “ How to conduct a bibliometric analysis: an overview and guidelines ”, Journal of Business Research , Vol. 133 , pp. 285 - 296 .

Gao , C. , Zeng , J. , Lo , D. , Xia , X. , King , I. and Lyu , M.R. ( 2022 ), “ Understanding in-app advertising issues based on large scale app review analysis ”, Information and Software Technology , Vol. 142 .

Gao , C. , Zeng , J. , Sarro , F. , Lo , D. , King , I. and Lyu , M. ( 2021 ), “ Do users care about ad's performance costs? Exploring the effects of the performance costs of in-app ads on user experience ”, Information and Software Technology , Vol. 132 , pp. 1 - 14 .

Ghorbani , Z. , Kargaran , S. , Saberi , A. , Haghighinasab , M. , Jamali , S.M. and Ale , E.N. ( 2022 ), “ Trends and patterns in digital marketing research: bibliometric analysis ”, Journal of Marketing Analytics , Vol. 10 No. 2 , pp. 158 - 172 .

Ghose , A. and Han , S. ( 2014 ), “ Estimating demand for mobile applications in the new economy ”, Marketing Science , Vol. 60 No. 6 , pp. 1351 - 1616 .

Graham , C. , Young , F. and Marjan , A. ( 2021 ), “ The generation Z audience for in-app advertising ”, Journal of Indian Business Research , Vol. 13 No. 3 , pp. 343 - 360 .

Green , M.C. and Brock , T.C. ( 2000 ), “ The role of transportation in the persuasiveness of public narratives ”, Journal of Personality and Social Psychology , Vol. 79 No. 5 , pp. 701 - 721 .

Grewal , D. , Bart , Y. , Spann , M. and Zubcsek , P.P. ( 2016 ), “ Mobile advertising: a framework and research agenda ”, Journal of Interactive Marketing , Vol. 34 , pp. 3 - 14 .

Haenschen , K. and Wolf , J. ( 2019 ), “ Disclaiming responsibility: How platforms deadlocked the federal election commission's efforts to regulate digital political advertising ”, Journal of Telecommunications Policy , Vol. 43 No. 8 , pp. 1 - 11 .

Handel , J. ( 2022 ), “ Back to basics: Mobile web vs. Mobile in-App advertising ”, available at: www.smaato.com/blog/back-to-basics-mobile-web-vs-mobile-in-app-advertising/ (accessed 24 February 2023 ).

Hao , L. , Guo , H. and Easley , R.F. ( 2016 ), “ A mobile platform's in-App advertising contract under agency pricing for app sales ”, Production and Operations Management , Vol. 26 No. 2 , pp. 189 - 202 .

Hsu , C.E. , Lopez , O. and Raj , Y.S. ( 2020 ), “ Considering parental perceptions on mobile learning systems with in-app advertising-Taiwan case study ”, International Journal of Advanced Science and Technology , Vol. 29 No. 7 , pp. 2437 - 2471 .

Hu , W. , Sun , X.-C. and Zhou , W.-F. ( 2023 ), “ Analysis of in-app advertising contract of a mobile platform and two competitive app developers ”, Journal of Industrial and Management Optimization , Vol. 19 No. 9 , pp. 6398 - 6429 .

Huh , J. ( 2017 ), “ Considerations for application of social science research approaches to digital advertising research ”, in Rodgers , S. and Thorson , E. (Eds), 3rd ed ., Digital Advertising: Theory and Research , Routledge , London , pp. 446 - 453 .

Interactive Advertising Bureau ( 2022 ), “ IAB Europe’s guide to in-app advertising ”, available at: www.iabeurope.eu/wp-content/uploads/2022/02/IAB-Europe-Guide-to-In-app-advertising.pptx.pdf (accessed 20 February 2023 ).

Jebarajakirthy , C. , Maseeh , H.I. , Morshed , Z. , Shankar , A. , Arli , D. and Pentecost , R. ( 2021 ), “ Mobile advertising: a systematic literature review and future research agenda ”, International Journal of Consumer Studies , Vol. 45 No. 6 , pp. 1258 - 1291 .

Ji , Y. , Wang , R. and Gou , Q. ( 2019 ), “ Monetization on mobile platforms: Balancing in‐app advertising and user base growth ”, Production and Operations Management , Vol. 28 No. 9 , pp. 2202 - 2220 .

Jiao , Y. , Tang , C.S. and Wang , J. ( 2022 ), “ An empirical study of play duration and in-app purchase behavior in mobile games ”, Production and Operations Management , Vol. 31 No. 9 , pp. 3435 - 3456 .

Kahneman , D. and Tversky , A. ( 1979 ), “ Prospect theory: an analysis of decision under risk ”, Econometrica , Vol. 47 No. 2 , pp. 263 - 291 .

Kaplan , M. ( 2021 ), “ In app Vs mobile web advertising: How to choose ”, available at: www.inmobi.com/blog/2022/09/07/in-app-vs-mobile-web-advertising-how-to-choose (accessed 20 February 2023 ).

Katz , E. , Blumler , J.G. and Gurevitch , M. ( 1973 ), “ Uses and gratifications research ”, Public Opinion Quarterly , Vol. 37 No. 4 , pp. 509 - 523 .

Keem , J.S. and Lee , S. ( 2018 ), “ A study on consumers’ experiences and avoidances of mobile shopping application advertisements ”, Journal of Global Fashion Marketing , Vol. 9 No. 2 , pp. 148 - 160 .

Lang , A. ( 2000 ), “ The limited capacity model of mediated message processing ”, Journal of Communication , Vol. 50 No. 1 , pp. 46 - 70 .

Lavie , N. , Hirst , A. , de Fockert , J.W. and Viding , E. ( 2004 ), “ Load theory of selective attention and cognitive control ”, Journal of Experimental Psychology: General , Vol. 133 No. 3 , pp. 339 - 354 .

Lee , S. , Zhang , J. and Wedel , M. ( 2021 ), “ Managing the versioning decision over an app’s lifetime ”, Journal of Marketing , Vol. 85 No. 6 , pp. 44 - 62 .

Leppäniemi , M. , Sinisalo , J. and Karjaluoto , H. ( 2006 ), “ A review of mobile marketing research ”, International Journal of Mobile Marketing , Vol. 1 No. 1 , pp. 30 - 40 .

Liu , H. and Liu , S. ( 2019 ), “ Considering in-App advertising mode, Platform-App channel coordination by a sustainable cooperative advertising mechanism ”, Sustainability , Vol. 12 No. 1 , pp. 1 - 28 .

Logan , K. ( 2017 ), “ Attitudes towards in-app advertising: a uses and gratifications perspective ”, International Journal of Mobile Communications , Vol. 15 No. 1 , pp. 26 - 48 .

Mandler , G. ( 1982 ), “ The structure of value: Accounting for taste ”, in Clark , M.S. and Fiske , S.T. (Eds), Affect and Cognition , Lawrence Erlbaum Associates , Hillsdale , pp. 3 - 36 .

Maseeh , H.I. , Jebarajakirthy , C. , Pentecost , R. , Ashaduzzaman , M. , Arli , D. and Weaven , S. ( 2021 ), “ A meta-analytic review of mobile advertising research ”, Journal of Business Research , Vol. 136 , pp. 33 - 51 .

Mattke , J. , Maier , C. , Reis , L. and Weitzel , T. ( 2021 ), “ In-app advertising: a two-step qualitative comparative analysis to explain clicking behavior ”, European Journal of Marketing , Vol. 55 No. 8 , pp. 2146 - 2173 .

Miller , G.A. ( 1956 ), “ The magical number seven, plus or minus two: Some limits on our capacity for processing information ”, Psychological Review , Vol. 63 No. 2 , pp. 81 - 97 .

Mouawi , R. , Imad , H.E. , Ali , C. and Ayman , K. ( 2019 ), “ Crowdsourcing for click fraud detection ”, EURASIP Journal on Information Security , Vol. 2019 No. 1 , pp. 11 - 28 .

Mukherjee , A. , Sundarraj , R.P. and Dutta , K. ( 2017 ), “ Apriori rule-based in-app ad selection online algorithm for improving supply-side platform revenues ”, ACM Transactions on Management Information Systems , Vol. 8 Nos 2/3 , pp. 1 - 28 .

Narang , U. and Shankar , V. ( 2019 ), “ Mobile marketing 2.0: state of the art and research agenda ”, Marketing in a Digital World, Review of Marketing Research , Vol. 16 , pp. 97 - 119 .

Nuwamanya , E. , Nuwasiima , A. , Babigumira , J.U. , Asiimwe , F.T. , Lubinga , S.J. and Babigumira , J.B. ( 2018 ), “ Study protocol: using a mobile phone-based application to increase awareness and uptake of sexual and reproductive health services among the youth in Uganda. A randomized controlled trial ”, Reproductive Health , Vol. 15 No. 1 , p. 216 .

Page , M.J. , McKenzie , J.E. , Bossuyt , P.M. , Boutron , I. , Hoffmann , T.C. and Mulrow , C.D. ( 2021 ), “ The PRISMA 2020 statement: an updated guideline for reporting systematic reviews ”, British Medical Journal , Vol. 372 No. 71 .

Park , S.S. and Park , B. ( 2020 ), “ Advertising on mobile apps versus the mobile web ”, Journal of Advertising Research , Vol. 60 No. 4 , pp. 381 - 393 .

Paul , J. and Criado , A.R. ( 2020 ), “ The art of writing literature review: What do we know and what do we need ”, International Business Review , Vol. 29 No. 4 , pp. 1 - 7 .

Paul , J. and Rosado-Serrano , A. ( 2019 ), “ Gradual internationalization vs born-global/international new venture models: a review and research agenda ”, International Marketing Review , Vol. 36 No. 6 , pp. 830 - 858 .

Paul , J. , Lim , W.M. , O’Cass , A. , Hao , A.W. and Bresciani , S. ( 2021 ), “ Scientific procedures and rationales for systematic literature reviews (SPAR-4-SLR) ”, International Journal of Consumer Studies , Vol. 45 No. 4 , pp. 1 - 16 .

Petty , R.E. and Cacioppo , J.T. ( 1986 ), “ The elaboration likelihood model of Persuasion ”, in Berkowitz , L. (Ed.), Advances in Experimental Social Psychology , Academic Press , New York, NY , pp. 123 - 205 .

Pine , J. and Gilmore , J. ( 1999 ), The Experience Economy: Work is Theatre and Every Business a Stage , Harvard Business School Press , Boston .

Qureshi , Q.A. , Noor , N.A. and Hassan , S. ( 2022 ), “ Consumer perceived value of collaboration vs. Incentives in the context of in-App ads: digital natives vs. Digital immigrants ”, International Journal of e-Collaboration , Vol. 18 No. 1 , pp. 1 - 22 .

Rafieian , O. and Yoganarasimhan , H. ( 2020 ), “ Targeting and privacy in mobile advertising ”, Marketing Science , Vol. 40 No. 2 , pp. 193 - 218 .

Rodgers , S. and Thorson , E. ( 2019 ), Advertising Theory , 2nd ed. Routledge , London , doi: 10.4324/9781351208314 .

Rodgers , S. and Thorson , E. ( 2000 ), “ The interactive advertising model: How users perceive and process online ads ”, Journal of Interactive Advertising , Vol. 1 No. 1 , pp. 41 - 60 .

Rosenkrans , G. and Myers , K. ( 2018 ), “ Optimizing location-based mobile advertising using predictive analytics ”, Journal of Interactive Advertising , Vol. 18 No. 1 , pp. 43 - 54 .

Rutz , O. , Aravindakshan , A. and Rubel , O. ( 2019 ), “ Measuring and forecasting mobile game app engagement ”, International Journal of Research in Marketing , Vol. 36 No. 2 , pp. 185 - 199 .

Scott , K.M. , Richards , D. and Londos , G. ( 2018 ), “ Assessment criteria for parents to determine the trustworthiness of maternal and child health apps: a pilot study ”, Health and Technology , Vol. 8 Nos 1/2 , pp. 63 - 70 .

Shao , R. , Rastogi , V. , Chen , Y. , Pan , X. , Gao , G. , Zou , S. and Riley , R. ( 2018 ), “ Understanding in-App ads and detecting hidden attacks through the mobile App-Web interface ”, IEEE Transactions on Mobile Computing , Vol. 17 No. 11 , pp. 2675 - 2688 .

Sigurdsson , V. , Menon , V.R. , Hallgrimsson , A.G. , Larsen , N.M. and Fagerstrom , A. ( 2018 ), “ Factors affecting attitudes and behavioral intentions toward in-app mobile advertisements ”, Journal of Promotion Management , Vol. 24 No. 5 , pp. 694 - 714 .

Singh , S. and Dhir , S. ( 2019 ), “ Structured review using TCCM and bibliometric analysis of international cause-related marketing, social marketing, and innovation of the firm ”, International Review on Public and Nonprofit Marketing , Vol. 16 Nos 2/4 , pp. 335 - 347 .

Snyder , H. ( 2019 ), “ Literature review as research methodology: an overview and guidelines ”, Journal of Business Research , Vol. 104 , pp. 333 - 339 .

Statista ( 2022 ), “ Mobile advertising spending worldwide 2007-2024 ”, available at: www.statista.com/statistics/303817/mobile-internet-advertising-revenue-worldwide (accessed 24 February 2023 ).

Statista ( 2023 ), “ Distribution of free and paid apps in the apple app store and google play as of march 2023 ”, available at: www.statista.com/statistics/263797/number-of-applications-for-mobile-phones/ (accessed 26 February 2023 ).

Stocchi , L. , Pourazad , N. and Michaelidou , N. ( 2022 ), “ Marketing research on mobile apps: past, present and future ”, Journal of the Academy of Marketing Science , Vol. 50 No. 2 , pp. 195 - 225 .

Su , M.-Y. , Wei , H.-S. , Chen , X.-Y. , Lin , P.-W. and Qiu , D.-Y. ( 2018 ), “ Using ad-related network behavior to distinguish ad libraries ”, Journal of Applied Sciences , Vol. 8 No. 10 , p. 1852 .

Sun , Z. , Dawande , M. , Janakiraman , G. and Mookerjee , V. ( 2017 ), “ Not just a fad: Optimal sequencing in mobile in-app advertising ”, Information Systems Research , Vol. 28 No. 3 , pp. 1 - 18 .

Sundar , S. , Jia , H. , Waddell , T. and Huang , Y. ( 2015 ), “ Toward a theory of interactive media effects (TIME) ”, in Sundar , S.S. (Ed.), The Handbook of the Psychology of Communication Technology , Wiley-Blackwell , NJ , pp. 47 - 86 .

Sung , E.C. ( 2020 ), “ Consumers’ responses to mobile app advertisements during holiday periods ”, Journal of Consumer Marketing , Vol. 37 No. 3 , pp. 341 - 352 .

Sung , E.C. ( 2021 ), “ The effects of augmented reality mobile app advertising: viral marketing via shared social experience ”, Journal of Business Research , Vol. 122 , pp. 75 - 87 .

Sung , E. , Han , D. and Choi , Y.K. ( 2022 ), “ Augmented reality advertising via a mobile app ”, Psychology and Marketing , Vol. 39 No. 3 , pp. 543 - 558 .

Thomas , J. and Harden , A. ( 2008 ), “ Methods for the thematic synthesis of qualitative research in systematic reviews ”, BMC Medical Research Methodology , Vol. 8 No. 1 .

Transfield , D. , Denver , D. and Smart , P. ( 2003 ), “ Towards a methodology for developing evidence-informed management knowledge by means of systematic review ”, British Journal of Management , Vol. 14 No. 3 , pp. 207 - 222 .

TrendMicro ( 2019 ), “ Mobile ad fraud schemes: How they work, and how to defend against them ”, available at: www.trendmicro.com:www.trendmicro.com/vinfo/us/security/news/mobile-safety/mobile-ad-fraud-schemes-how-they-work-and-how-to-defend-against-them (accessed 1 March 2023 ).

Truong , V. ( 2023 ), “ Optimizing mobile in-app advertising effectiveness using app publishers-controlled factors ”, Journal of Marketing Analytics , doi: 10.1057/s41270-023-00230-w .

Truong , V. , Nkhoma , M. and Pansuwong , W. ( 2019 ), “ An integrated effectiveness framework of mobile in-app advertising ”, Australasian Journal of Information Systems , Vol. 23 , doi: 10.3127/ajis.v23i0.1971 .

Upshot.ai ( 2022 ), “ How banks can use mobile banking apps to cross-sell and upsell ”, available at: www.upshot.ai/blog/how-banks-can-use-mobile-banking-apps-to-cross-sell-and-upsell/ (accessed 1 March 2023 ).

Varnali , K. and Toker , A. ( 2010 ), “ Mobile marketing research: the-state-of-the-art ”, International Journal of Information Management , Vol. 30 No. 2 , pp. 144 - 151 .

Wang , S.S. and Chou , H. ( 2019 ), “ Effects of game‐product congruity on in‐app interstitial advertising and the moderation of media‐context factors ”, Psychology and Marketing , Vol. 36 No. 3 , pp. 229 - 246 .

Yang , Y. and Cao , G. ( 2018 ), “ Prefetch-based energy optimization on smartphones ”, IEEE Transactions on Wireless Communications , Vol. 17 No. 1 , pp. 693 - 706 .

Acknowledgements

The authors are grateful to the anonymous reviewers for their comments and suggestions. The authors express their gratitude to Prof. Carlos Flavian, Editor-in-Chief of the Spanish Journal of Marketing – ESIC for his constructive feedback to enable the publication of this paper. The authors are also thankful to the Manipal Universal Press for their editorial service.

Corresponding author

Related articles, we’re listening — tell us what you think, something didn’t work….

Report bugs here

All feedback is valuable

Please share your general feedback

Join us on our journey

Platform update page.

Visit emeraldpublishing.com/platformupdate to discover the latest news and updates

Questions & More Information

Answers to the most commonly asked questions here

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

Enter the email address you signed up with and we'll email you a reset link.

• We're Hiring!
• Help Center

The effectiveness of advertising: a literature review

The aim of the paper is to propose a simple approach to monitor and control the encoding phase, too often neglected by the doctrine and operators in the communication process. The paper is discusses a possible methodology to investigate the degree of coincidence / discordance between what is conveyed through the advertising and what is actually understood by some consumers. The object of investigation is the verbal text of the spot in order to verify the level of understanding of the message and, if necessary, to find the reasons of a wrong perceived meaning.

RELATED PAPERS

nationalacademies.org

Peter Kretschmar

Handbook of Research on Artificial Intelligence Techniques and Algorithms

Nuno Garcia

Roland Taremwa

Macromolecules

Peter Thune

Nasza Przeszłość

Grzegorz Kublin

American Journal of Engineering and Applied Sciences

Taher Abu lebdeh

Jurnal Teknik Mesin dan Pembelajaran

rahmad syah

Acta Horticulturae

françois Léger

Journal of Physics and Chemistry of Solids

Vipul Srivastava

Astroparticle Physics

Robert Mutel

DYNA INGENIERIA E INDUSTRIA

Angel Miguel Celi Rodriguez

Journal of Plant Process and Function

Fatemeh Daneshmand

Aina Tatheer

Ika Rifqiawati

PRIMA: Journal of Community Empowering and Services

Joko Winarno

Lautaro Colautti

Owais Ahmad

Spinal Cord

A. Bjerkefors

Arminda Lopes

Gene Therapy

Kathleen Pirollo

Hacettepe journal of mathematics and statistics

Javanshir Hasanov

Euro surveillance : bulletin Europeen sur les maladies transmissibles = European communicable disease bulletin

Miguel Martinez

Slovo Journal of Slavic Languages and Literatures

Thomas Rosén

Frontiers in Psychology

Richard M Ryan

Deo Z Namwira

See More Documents Like This

RELATED TOPICS

•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024

Advertising appeals effectiveness: a systematic literature review

Affiliation.

• 1 Social Marketing, Griffith University, 170 Kessels Road, Nathan, QLD 4111, Australia.
• PMID: 34931233
• DOI: 10.1093/heapro/daab204

Positive, negative and coactive appeals are used in advertising. The evidence base indicates mixed results making practitioner guidance on optimal advertising appeals difficult. This study aims to identify the most effective advertising appeals and it seeks to synthesize relevant literature up to August 2019. Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses framework a total of 31 studies were identified and analyzed. Emotional appeals, theory utilization, materials, results and quality were examined. Across multiple contexts, results from this review found that positive appeals were more often effective than coactive and negative appeals. Most studies examined fear and humour appeals, reflecting a literature skew towards the two emotional appeals. The Effective Public Health Practice Project framework was applied to assess the quality of the studies and identified that there remains opportunity for improvement in research design of advertising studies. Only one-third of studies utilized theory, signalling the need for more theory testing and application in future research. Scholars should look at increasing methodological strength by drawing more representative samples, establishing strong study designs and valid data collection methods. In the meantime, advertisers are encouraged to employ and test more positive and coactive advertising appeals.

Keywords: advertising appeals; advertising appeals effectiveness; emotional appeals; systematic literature review.

© The Author(s) 2021. Published by Oxford University Press. All rights reserved. For permissions, please email: [email protected].

Publication types

• Systematic Review
• Advertising*
• Public Health Practice

Assessing Religious Leadership: A Scoping Review of Leadership Effectiveness Criteria and Current Trends in the Academic Literature

• Open access
• Published: 15 November 2023
• Volume 73 , pages 1–21, ( 2024 )

Cite this article

You have full access to this open access article

• Annemarie Foppen   ORCID: orcid.org/0000-0002-8527-8540 1 &
• Joke W. van Saane 2  

1315 Accesses

3 Altmetric

Explore all metrics

A clear understanding of what constitutes effective religious leadership in the context of faith communities is essential for religious practitioners, religious communities, and educational institutes. Twenty-five years after the latest review study by Nauss, an updated overview is needed to account for new insights, especially regarding the latest developments in leadership research and the changing religious landscape. A scoping review was conducted in four databases from 1997 to 2022 to examine the academic literature. The search identified 64 unique references that researched effective religious leadership using 27 different effectiveness criteria. These criteria were categorized based on their target of evaluation: (1) the religious leader, (2) followers/members, and (3) the congregation. Although most references assessed leadership effectiveness by evaluating the religious leader, the most used effectiveness criterion was ‘attendance or numerical growth.’ Over the last 25 years, the criteria for religious leadership effectiveness have become more diverse, and a growing number of references combine criteria from multiple targets to evaluate effectiveness. However, the focus on Christian leadership remains prevalent in the academic literature. The present overview is intended as a starting point for future research as it identifies the current trends and existing knowledge gaps. The study’s findings also invite religious practitioners and congregations to reflect on their methods of assessing leadership effectiveness.

Similar content being viewed by others

Epilogue: What Shall We Do Now?

A systematic review of leader credibility: its murky framework needs clarity

Ralph I. Williams Jr, Deana M. Raffo, … Leigh Anne Clark

Analyzing Leadership Attributes in Faith-Based Organizations: Idealism Versus Reality

Krystin Zigan, YingFei Héliot & Alan Le Grys

Avoid common mistakes on your manuscript.

Introduction

Assessing religious leadership is important for religious practitioners, religious communities, and educational and religious institutes. Insight into which competencies make religious leaders effective and how they affect people’s lives and their congregations can result in more vibrant religious communities (Boyatzis et al., 2011 ). However, identifying the criteria for religious leadership effectiveness has proven difficult (Malony, 2000 ; Nauss, 1994 ). The measurement of effectiveness among religious leaders is complicated and, in some contexts, debated or even resisted (McKenna & Eckard, 2009 ). Some consider religious leadership effectiveness untraceable due to its spiritual or transcendental dimension (Malony, 1984 ), making it not objectively measurable. For others, existing methods of assessment are perceived as uninspiring or unsuitable for their congregation (e.g., numerical growth; Dadswell & Ross, 2013 ; Strunk et al., 2017 ). The diverse contexts in which religious leaders operate provide another challenge. What is considered effective religious leadership can vary depending on the context, religion, denomination, congregational size, and geographical location (Nauss, 1996 ). Just as leadership has been defined in many ways, conceptions of leadership effectiveness vary among scholars; as Yukl and Gardner ( 2020 ) state, “The selection of [leadership effectiveness] criteria depends on the objectives and values of the person making the evaluation” (p. 30). As there does not seem to be a shared definition of leadership effectiveness in the literature nor in the domain of religion, and as most references studied were not explicit about their definition, the current study aims to provide an overview of the various effectiveness criteria used to assess religious leadership in the recent academic literature. Instead of proposing our own definition of effective religious leadership, we intend to identify and clarify the different ways effective religious leadership is being evaluated, sorted by the target of evaluation.

DeRue et al. ( 2011 ) propose a model of leader traits, behaviors, and effectiveness in which leadership effectiveness criteria are categorized along three dimensions to organize the various effectiveness criteria. The model differentiates between effectiveness criteria based on their (1) content (i.e., task performance, affective and relational criteria, or overall judgments of effectiveness), (2) the level of analysis (i.e., individual, dyadic, group, or organizational), and (3) the target of evaluation (e.g., leader effectiveness, group performance).

In his review study, Nauss ( 1996 ) utilizes the latter as he identifies two types of criteria used for evaluating clergy effectiveness based on their target of evaluation, namely, the minister and the congregation. The first type, referred to as primary criteria, includes descriptions of the behavior and characteristics of the minister, such as ‘preaching behavior,’ ‘ability to serve,’ ‘personal characteristics,’ and ‘traits.’ The second type, i.e., secondary criteria, focuses on the effects or consequences of the minister on the congregation and its members, such as ‘church growth,’ ‘commitment,’ ‘satisfaction,’ and ‘changes in faith.’ To the best of our knowledge, Nauss's review of the empirical literature remains, to this day, the most comprehensive overview of assessing effective religious leadership in the academic literature. However, the kind of leadership that is considered effective changes over time. Social, cultural, and technological changes have significantly impacted religious practice, causing changes in our perceptions of the behaviors and characteristics that contribute to religious leaders’ effectiveness (Dobrotka, 2021 ). Thus, after 25 years, the field requires an updated overview to account for new insights, especially concerning the latest developments in leadership research and the changing religious landscape.

A changing landscape

Global mobility has changed the religious landscape radically from the 20th to the 21st century as millions of migrants have brought their religions, values, and traditions to other parts of the world (Giordan, 2014 ). Consequently, many nations transformed from homogeneous religious societies into places with much religious diversity. The work of Nauss ( 1996 ) and the empirical studies he reviews, however, are limited to Christian leadership, in most cases Protestant ministers. The social scientific study of religion traditionally also has a narrow focus on Christianity and Judaism. However, other religions have gradually been included, such as Hinduism, Buddhism, Islam, and new religious movements (Rambo & Haar Farris, 2012 ). For the current article, we aimed to take into account the growing variety of religions and traditions by not focusing on a specific religion in our data search.

Additionally, almost every high-income country faces lower levels of religiosity, and many other countries have become less religious as well (Inglehart, 2021 ). This trend is reflected in the declining number of individuals involved in organized religion in Western societies and attending religious services (Brauer, 2018 ; European Social Survey, 2002 , 2016  as cited in De Hart & Van Houwelingen, 2018 ). At the same time, a rise in post-Christian or New Age spirituality is observed, characterized by a more individualized, experience-oriented, and anti-institutional spirituality (Marshall & Olson, 2018 ; Tromp et al., 2020 ). Along with these developments, confidence in and obedience to traditional religious leaders has declined in these societies (Hoffmann, 2013 ; Inglehart, 2020 ), giving rise to new types of authority and leadership. These changes in the religious landscape present new challenges for religious leaders in the West and will undoubtedly impact how effective religious leadership is viewed and evaluated.

Over the past decades, academic thinking about leadership has also changed (Zhu et al., 2019 ). In line with most of the earlier studies on ministerial effectiveness, Nauss focused on the religious leader and determined effective religious leadership in terms of ministerial functions or activities (McKenna & Eckard, 2009 ; Nauss, 1972 , 1996 ). This approach to leadership can be categorized as part of what Haslam et al. ( 2015 ) call “the classical leadership perspective.” New insights in leadership research, however, suggest that leadership is not solely about the person of the leader (e.g., character and competencies) but rather should be conceived of as a relational process between the leader, the followers, and the group (Haslam et al., 2020 ; Steffens et al., 2021 ). Leadership is now generally studied as a process of influencing other people to accomplish shared goals (Wilson, 2022 ). In line with this, most researchers evaluate leadership effectiveness “in terms of the consequences of influence on an individual, group, or organization” (Yukl & Gardner, 2020 , p. 28).

The current study

To account for these changes and developments in the religious landscape and leadership research, we conducted a scoping review in the recent academic literature to provide an overview of the various effectiveness criteria used to assess religious leadership over the last 25 years. A systematic approach was employed to search for studies on religious leadership that explicitly address the issue of effectiveness. A combination of search terms was used to account for the variety of religions (i.e., Abrahamic faiths, Hinduism, Buddhism, and spirituality), the various terms for effectiveness (e.g., success, flourishing, thriving, vitality, vibrancy), and the various indicators for religious leaders (e.g., clergy, minister, pastor, imam, priest, rabbi). However, to limit ourselves, we focused exclusively on what Chaves et al. ( 2022 ) call congregations’ religious leaders: “religious specialists who are leaders of organized religious communities.” As a result, this review did not examine religious or spiritual leaders in more diffuse and fluid, interstitial settings (Ammerman, 2020 ), such as spiritual guides or meditation teachers; nor religious or spiritual leadership in secular or organizational contexts, such as spiritual caregivers in health institutions (Timmins et al., 2018 ); or organizational leaders who incorporate spiritual values in the workplace (Oh & Wang, 2020 ). In addition, this review concentrated solely on the academic literature (i.e., peer-reviewed articles, books, and dissertations) as this literature could be accessed and searched systematically through bibliographic databases. Also, this approach ensures that only studies meeting the highest scholarly quality standards are included. Consequently, we did not evaluate the professional literature and the extensive work being done outside of academia, such as Duke’s Faith and Leadership initiative. We strongly encourage others to follow up on the current study by reviewing this body of literature as well.

The purpose of the current study is to contribute to the field by building on the work of Nauss ( 1996 ) and extending it in three ways: (1) by updating the data and mapping religious leadership effectiveness criteria that have been used over the last 25 years, (2) by charting recent trends and developments in the academic literature, and (3) by broadening the search from an emphasis on Protestant Christian ministers to reviewing effective religious leadership across different religions. The studies’ findings and conclusions are used to propose directions for future research on the topic.

The present study used a scoping review methodology to examine the academic literature on effective religious leadership. This scoping review aimed to survey the academic literature on effective religious leadership, identify the criteria used to evaluate effectiveness, and detect any knowledge gaps (see Munn et al., 2018 ). We adhered to the five steps of Arksey and O’Malley’s ( 2005 ) methodological framework for conducting a scoping study—(1) identifying the research question, (2) identifying relevant studies, (3) selecting the studies, (4) charting the data, (5) and compiling, summarizing, and reporting the results (Levac et al., 2010 ). The research question that guided this scoping review is: What leadership effectiveness criteria are used in the recent academic literature to assess religious leadership?

Identifying relevant studies

The review of Nauss ( 1996 ), which assessed relevant studies on effective religious leadership up to and including 1996, served as a reference point. A comprehensive search was performed in the bibliographic databases Atla Religion Database, PsycInfo, Eric, and Psychology and Behavioral Sciences Collection (via Ebsco) from January 1, 1997, to January 12, 2022. To conduct the search, we first combined various terms for ‘religion’ and ‘leaders(hip)’ to search for leadership studies across various religions. In addition, all studies with direct synonyms for religious leaders were searched for using terms such as ‘clergy,’ ‘pastor,’ ‘minister,’ ‘rabbi,’ and ‘imam.’ Next, a focus on effectiveness was added by combining the search string above with the term ‘effectiveness’ and closely related indicators such as ‘success,’ ‘flourishing,’ ‘thriving,’ ‘vital,’ ‘growing,’ ‘vibrant,’ and ‘efficacy.’ The full search strategy is presented in Table 1 . From these results, we included journal articles, dissertations, and books in the final search. The search was performed without language restrictions, and duplicates were excluded. The database search identified a total of 11,653 references.

Study selection

The two-stage review process was divided into a title-and-abstract screening and a full-text screening round. The 11,653 references were submitted to Rayyan (rayyan.qcri.org), an online review application, to facilitate the review process. References were included when they studied congregations’ religious leaders, defined in this study as “religious specialists who are leaders of organized religious communities,” as opposed to religious specialists who are private practitioners (Chaves et al., 2022 , p. 737). Next, to be included in this study, references had to explicitly address the issue of effectiveness either as a focus of empirical research or theoretical reflection. The first author screened the articles and discussed her findings midway with the second author to further refine the inclusion and exclusion criteria. In addition, the second author reviewed a random selection of articles to assess the reliability of the review process ( n  = 112), resulting in an interrater agreement of 71% regarding which studies to include or exclude.

A total of 11,541 references were excluded during the abstract-and-title screening because they did not meet the inclusion criteria above, because they did not meet the initial criteria (i.e., not being published as a journal article, dissertation, or book), because an abstract was not available ( n  = 17), or because an abstract was not available in English and the reference was written in a language other than English, Dutch, French, or German ( n  = 2). Next, a full-text screening was performed with the remaining 112 references. During this phase, it was decided to focus only on journal articles and dissertations as obtaining the full text of the seven remaining books was challenging. Another 40 references were excluded based on the inclusion criteria during the full-text evaluation. One reference was excluded because the authors of the current study were not proficient in Polish. See Fig.  1 for the flow diagram and the number of references included in this review.

Flow Diagram of the Screening Process

Charting the data

For each of the 64 references included in this study, an overview was created with general information about the study. This overview included the study’s author(s), year of publication, research question, theoretical embedding, research methodology, the method used to assess effective religious leadership, and the most important outcomes reported in the study’s abstract. Charting how effective religious leadership was defined across the different studies proved difficult as many references did not provide an explicit definition. After careful consideration, it was decided to focus on recording the specific criteria used to assess religious leadership effectiveness.

Twenty-six of the 64 references examining effective religious leadership that were included in this study were published in 21 different academic journals, while the other 38 studies were dissertations. Ten studies were theoretical, literature-based contributions, whereas the additional 54 were empirical (i.e., 22 quantitative studies, 17 qualitative studies, and 15 mixed-method studies). Although the studies varied with regard to their discipline (e.g., psychology, religious studies, theology) and motives (to describe, understand, predict, or prescribe), there was a strong focus on Christian leadership ( n  = 60), with only three studies investigating Muslim leadership and one study researching religious leadership in general. In the years after 2010, the number of references published each year was considerately higher, with an average of 2.3 before 2010 and 3.5 from 2011 onwards. The nationality of the authors varied only minimally. All the dissertations were submitted to American universities, and the first author was situated in the United States in 16 of the 26 journal articles. The other 10 references were written by scholars from Australia (2), Nigeria (2), Germany (1), Indonesia (1), Ireland (1), Malaysia (1), South Africa (1), and the United Kingdom (1). An overview of the descriptives can be found in Table 2 .

Religious leadership effectiveness criteria

Twenty-seven different criteria of religious leadership effectiveness were identified among the included references. The criterion used most often in the references to assess effective religious leadership was ‘attendance or numerical growth’ (20 out of the 64 references; 31%). Other criteria that were used in ten or more of the references were ‘personal spirituality/character’ (11 references; 17%), ‘leadership style’ (10 references; 16%), ‘leadership skill and qualities’ (10 references; 16%), ‘perceived effectiveness’ (10 references; 16%), and ‘organizational mission and goals’ (10 references, 16%). Although we observed some differences between the journal articles and the dissertations, the overall use of criteria was quite similar, with an overlap of 23 out of the 27 identified criteria (89%). The most significant difference was observed in the criteria of ‘personal(ity) traits’ and ‘numerical growth or attendance’ as they were used as a criterion in 8 and 14 dissertation studies, respectively, versus 1 and 6 journal articles.

Following the model by DeRue et al. ( 2011 ), we categorized the leadership effectiveness criteria by their target of evaluation, which refers to “whether the leader is the target of evaluation (e.g., leader effectiveness, satisfaction with leader) or another outcome that is within the domain of leadership effectiveness but not specific to the leader [such as] group performance” (p. 11). The criteria used to assess religious leadership effectiveness could be subdivided into three target groups of evaluation:

leader -focused criteria that assess characteristics and outcomes associated with the religious leader as the target of evaluation, such as leadership style, leadership skills, or perceived effectiveness;

follower/member -focused criteria that evaluate outcomes specific to the followers or members of the congregations, such as satisfaction or spiritual growth;

congregation -focused criteria of effectiveness that evaluate outcomes associated with the religious community, such as numerical growth or attendance and organizational mission and goals.

An overview of the criteria sorted by target group can be found in Table 3 .

Leader-focused criteria

References that used leader-focused criteria examined individual characteristics, traits, skills, and qualities of the religious leader to determine effectiveness. The most utilized effectiveness criteria were ‘personal spirituality/character’ and ‘leadership style.’ The leadership styles associated with effective leadership in these references center on the relationship between leader and followers, such as transformational leadership (Taylor, 2019 ; Wasberg, 2013 ), servant leadership (Agee, 2001 ; Graham-Brown, 2020 ), and relational leadership (Watt, 2014 ). Other references list specific leadership skills, qualities, or personal traits as characteristic of effective religious leadership. The Ministerial Effectiveness Inventory (MEI; Majovski, 1982 ) was used in three references to assess leaders’ ministerial competencies; it contains items such as ‘the minister enables the congregation to experience opportunities for personal growth and spiritual enrichment’ and ‘I have an approach to ministry that emphasizes evangelistic and mission goals.’

Follower/member-focused criteria

The effectiveness criteria in this target group center around the congregants as the primary focus of leadership evaluation. Some criteria that focus on followers or members include their satisfaction with leadership, worship, and sacraments; how well their needs are being met; and their sense of community and financial support to the congregation. The most common follower/member-focused criterion used to measure leadership effectiveness was ‘spiritual growth.’

Congregation-focused criteria

The congregation-focused criteria operationalized leadership effectiveness through its effect on the religious community. The criterion most frequently used with respect to the congregation was’numerical growth/attendance.’ Other numerical criteria in this category included the number of conversions, baptisms, church plants, and religious classes. Additionally, references focused on the achievement of ‘organizational mission and goals,’ the degree of ‘community service and outreach,’ and congregational health / well-being as indicators of leadership effectiveness in the congregation.

Ways of assessing effective religious leadership

We observed different approaches to assessing effective religious leadership in the included references based on the targets that were used for evaluation. Some studies assessed religious leadership solely by using leader-focused criteria; other references exclusively used follower/member-focused or congregation-focused criteria. A last group combined the three target groups in various ways to determine leadership effectiveness (i.e., multiple targets). Most references utilized leader-focused criteria to determine leader effectiveness (44%), but a significant number of references used a multiple-targets approach to assess effective religious leadership (34%). This approach was especially prevalent in the dissertation studies. These references tended to take a more comprehensive approach to assessing religious leadership (e.g., Boyatzis et al., 2011 ; McKenna & Eckard, 2009 ). Among the journal articles, the focus on leader indicators was particularly prevalent (see Table 4 for an overview).

Trends and developments in the field

Indicators of effectiveness: from a focus on the minister’s behavior to a more diverse pallet.

While the main focus in Nauss’s review ( 1996 ) was on the minister and their observable behavior, more recent studies also include followers and the congregation as a factor in their evaluation of leadership effectiveness, with ‘numerical growth or attendance’ being the most used criterion of effectiveness. However, the centrality of leader-focused criteria still appears in the recent literature, especially in journal articles. At the same time, the pallet of indicators has become more diverse. For instance, ‘leadership style’ and ‘personal spirituality/character’ were added to the list of indicators, as well as some follower/member- and congregation-focused criteria such as meeting ‘followers’ needs,’ ‘congregational health,’ and achieving ‘organizational mission and goals.’

Beyond the bifocal view: Interactions between leader and congregation

In 1996, Nauss observed a bifurcation in his review study on how ministerial effectiveness was defined. References either focused “upon the minister’s behavior or the effects of the minister’s work in the activities and behavior of the church and its members.” A significant number of the references included in the current study departed from this bifocal view on assessing effective religious leadership by adopting a more dynamic approach. Twenty-two references explicitly combined criteria from multiple target groups to determine leadership effectiveness (5 articles and 17 dissertations). Also, many references that assessed effectiveness by focusing on one specific target often took the interactions between the leader and the congregation into account. For example, in various studies, leader characteristics were used as a predictor of follower/member-focused or congregation-focused criteria of effectiveness (e.g., Adams, 2013 ; Al-Hilali, 2012 ; Cobb-Hayes, 2018 ; Oney, 2009 ; Rowold, 2008 ; Rumley, 2011 ; Soldo, 2017 ). In addition, the transformational leadership model, a follower-oriented approach, was a recurring theory used by about 20 references (30%), either in the study’s design or as part of the theoretical embedding.

The “ultimate criterion”: A value-based approach to assessing religious leadership persists

A value-based approach to assessing religious leadership was recognized across the references as dominant criteria of effectiveness included ‘personal spirituality/character,’ leadership styles such as servant leadership, ethical leadership, and spiritual leadership, ‘organizational mission and goals,’ and ‘spiritual growth.’ Spiritual indicators were already being used as empirical measures of effectiveness in the review by Nauss ( 1996 ), but only minimally.

Measures of effective religious leadership remain scarce

Based on the included references, it appears that hardly any new instruments have been developed to measure effective religious leadership over the past 25 years, and there seems to be a specific lack of validated measures. The Ministerial Effectiveness Inventory by Majovski ( 1982 ) was used three times to map ministerial competencies. Another nine references administered the Multifactor Leadership Questionnaire (Avolio & Bass, 2004 ), a psychological inventory that measures transformational, transactional, and laissez-faire leadership and three leadership outcomes (i.e., followers’ perception of leadership effectiveness, followers’ satisfaction with the leadership, and the leader’s ability to meet followers’ needs, inspire extra effort, and achieve organizational goals). In addition, studies that used congregation-focused criteria indicators to assess leadership effectiveness sometimes employed congregational models, such as the National Church Live Survey and the Natural Church Development characteristics (Powell et al., 2022 ; Schwarz, 2006 ).

The focus on Christian leadership remains

Only four references were included in this study that examined effective religious leadership outside the Christian context (6%). One study discussed religious leadership in general (Malony, 2000 ), and three studies examined Muslim leadership (Al-Hilali, 2012 ; Mahazan & Abdullah, 2013 ; Soldo, 2017 ). Given the few references that examined religious leadership outside the Christian context, there still appears to be a one-sided emphasis on Christian leadership in the existing literature. Although, at first glance, the ways of assessing religious leadership in these four studies do not appear to diverge from the main findings discussed above, we cannot generalize our findings on religious leadership effectiveness outside of the Christian context.

The present study conducted a scoping review to provide an up-to-date overview of how effective religious leadership has been assessed over the last 25 years in the academic literature and to chart recent trends and developments in the field. A literature search yielded 64 references that together used 27 different criteria to assess religious leadership effectiveness. The criteria were categorized based on their target group of evaluation, namely, leader, follower/member, or congregation. Most references used leader-focused criteria to assess religious leadership, but a significant number of studies also employed a multiple-targets approach to determine leadership effectiveness. Comparing our results to the review conducted by Nauss ( 1996 ), we did not observe drastic changes in how effective religious leadership is assessed. There were, however, some significant developments, and these are discussed below.

Next to the importance of leader-focused criteria, follower/member-focused and congregation-focused criteria are increasingly being included as valid measures of effectiveness. ‘Attendance or numerical growth’ was the criterion most frequently used in the included references to assess effectiveness. Although congregational growth has been debated as a criterion of vitality or effectiveness (e.g., Thiessen et al., 2019 ), success and status are often attributed to those leaders who can break with the downward trend and draw large numbers of visitors to their services. Linking congregational growth or decline to leadership efforts aligns with several studies showing that clergy contribute to church growth (e.g., Francis et al., 2015 ; Haskell et al., 2016 ; Wollschleger, 2018 ).

Recent references adopted a more dynamic and extensive approach to assessing effective religious leadership by combining multiple criteria and different target groups. The shift from a strong focus on the leader to a more follower- and group-oriented approach to religious leadership has taken longer than it has in the mainstream research on leadership (Haslam et al., 2020 , pp. 19–44) but appears to be prevalent in the current literature on religious leadership as well. Transformational leadership has a prominent place in the references in the current review study, as it does in most contemporary leadership research (Dinh et al., 2014 ; Zhu et al., 2019 ), which shows a continued interest in (neo)charismatic leadership topics such as inspirational leadership, visionary leadership, empowerment, and followers’ trust. Transformational leadership refers to a multidimensional leadership style in which leaders motivate followers to perform beyond expectations by transforming followers’ attitudes, beliefs, and values as opposed to simply gaining compliance (Bass, 1985 ; Rafferty & Griffin, 2004 ; Yukl, 1999 ). Behavior linked to transformational leadership has the following four components (Bass & Riggio, 2005 ): idealized influence (i.e., serving as a role model for followers), inspirational motivation (i.e., communicating a stimulating vision), intellectual stimulation (i.e., encouraging innovation and creativity), and individualized consideration (i.e., attention for followers’ development).

A value-based approach to assessing religious leadership remained prevalent in the reviewed body of research, with many references using indicators such as ‘personal spirituality/character,’ ‘servant leadership,’ and ‘spiritual growth’ to determine effectiveness. A normative approach to assessing religious leadership also appears inherent to researching leadership in the religious context (McKenna & Eckard, 2009 ; Wollschleger, 2018 ). The spiritual dimension entails a normative element to effective religious leadership, resulting in ultimate criteria of effectiveness based on specific beliefs or values that are so important that they are usually not open for debate or modification. Although such a prescriptive approach to leadership appears specific to religion, value-based approaches to leadership are also emerging in mainstream research on leadership (Dinh et al., 2014 ; Zhu et al., 2019 ). Recent studies demonstrate a growing interest in leadership behaviors that include value elements such as ethical leadership, authentic leadership, spiritual leadership, and servant leadership. Zhu et al. ( 2019 ) link this trend to a growing focus on corporate social responsibility.

Implications for research and practice

By identifying the various criteria of effectiveness and sorting the literature into three target groups for assessing religious leadership, this study offers clarity and coherence in a rather fragmented field of research. Consequently, this overview can function as a starting point for future research identifying the trends and developments in the field and current knowledge gaps. This will make it easier for researchers to determine their research focus and build on previous work. Some concrete recommendations for future research are given below. Additionally, the current findings show interesting leads for collaboration between researchers of religious leadership and mainstream leadership scholars. For example, the shared interest in transformational leadership would be an interesting starting point as this theory has some evident religious connotations (Spoelstra, 2017 ). Aside from its focus on charisma, which has historical roots in Christian writings, common elements include the central notion of transformation and the religious concept of conversion. The transformational leader is expected to induce change within their followers to transform (i.e., convert) them into more effective, engaged, or moral individuals, embodying higher values for the company’s good (Delaney & Spoelstra, 2022 ; Spoelstra, 2017 ). Another lead for collaboration could be the emergence of value-based leadership in mainstream leadership research (Zhu et al., 2019 ) as a value-based approach to leadership is central in the domain of religion as well (e.g., Agee, 2001 ; Graham-Brown, 2020 ; Taylor, 2019 ).

The current findings invite religious practitioners and congregations to reflect on their own ways of evaluating leadership effectiveness. What are current practices, and what would be appropriate ways to measure effectiveness based on the shared beliefs and values of their specific contexts? The overview of indicators of effectiveness used in recent research will hopefully inspire this process and encourage new and more comprehensive approaches to measure effectiveness in helping religious communities to flourish. Educational institutes can benefit from the current findings by reflecting on the trends and developments that have been outlined. The growing significance of follower-oriented approaches to leadership can, for example, lead to new areas of focus in educational programs and the screening of candidates for religious vocations.

Limitations

A limitation of the current study design is that the outcomes of this review study are confined to the references that were included based on the data search that was conducted. In our discussion of the literature, we stayed close to the included references and focused on the trends and insights that emerged from these specific references. As a result, we might have missed relevant works on the study of religious leadership effectiveness because they were outside of our search terms and inclusion criteria. For example, the reviewers have brought to our attention the work of Friedman ( 2007 , 2011 ). His work was widely influential in the 2000s and has been especially valued by mainline Protestant and Jewish leaders for its holistic approach to religious leadership based on Bowen’s family systems theory. Friedman’s work was most likely not identified during our literature search because it does not explicitly address religious leadership and partly falls outside the selected timeframe. The same will apply to the professional literature as our data search was limited to the academic literature. In addition, studies that did not research religious leadership in the context of faith communities or that did not explicitly discuss the issue of effectiveness also fell outside the focus of the current study.

Recommendations: An agenda for future research and practice

A theme that should be high on the research agenda is broadening the scope of research from focusing on Christian leadership to examining religious leadership effectiveness across different religions. The current data search only identified three references that researched religious leadership from a Muslim perspective and one reference that focused on religious leadership in general. This suggests that the findings of the current study may not be applicable beyond the Christian context as only limited academic research was included on the effectiveness of religious leadership in general. Therefore, future research should focus on conducting studies across various religious backgrounds to deepen our understanding of effective religious leadership. Additionally, the studies included in the current overview were conducted mainly by scholars from the United States examining religious leaders in an American context. We therefore also need more studies from underrepresented parts of the world to get a more comprehensive perspective on religious leadership effectiveness.

Another recommendation for future research concerns the coherence in the field and its theoretical foundation. The references included in this study do not strongly build on previous work in the field, and, as a consequence, research on religious leadership appears to be less incremental. Also, an overarching narrative of assessing effective religious leadership seems to be lacking as academic reflection on the topic stagnated after the work of Nauss and Malony (e.g., Malony, 1984 , 2000 ; Malony & Hunt, 1991 ; Nauss, 1972 , 1996 ). An updated theoretical foundation that does justice to the current landscape is essential for the field to progress. Future studies can build on such a foundation and create more sound evidence regarding religious leadership effectiveness by collecting data that supplements earlier findings and addressing current knowledge and research gaps. For example, a theoretical framework could combine the three target groups (i.e., leader, follower, and congregation) into a dynamic and comprehensive model (see Boyatzis et al., 2011 ). Additionally, transformational leadership could serve as an overarching theory given the popularity it already has in the field and the promising effects of this type of leadership style on the congregation (e.g., Carter, 2009 ; Rowold, 2008 ).

A concrete recommendation for practice that follows from the current overview is to use multiple criteria from multiple target groups to evaluate religious leadership in congregations (DeRue et al., 2011 ). As McKenna and Eckard ( 2009 ) point out, the criteria deemed appropriate will vary depending on the mission and overall purpose of the congregation. However, using a range of criteria can help overcome a one-sided focus on, for example, numerical growth or leadership competencies. By evaluating religious leadership using various criteria, we can better understand the dynamic and diverse leadership contexts and, hopefully, alleviate some of the discomfort people feel in assessing religious leadership effectiveness.

A theme for the research agenda that follows from the above is the need for reliable and validated scales to measure religious leadership effectiveness. Over the past 25 years, only a few new instruments have been developed. One possible reason for this could be the limited interest of social scientists in religious leadership. The only validated measure used by references in this study that was specifically designed for the assessment of religious leadership was the Ministerial Effectiveness Inventory by Majovski ( 1982 ), a shortened version of the Profiles of Ministry (Schuller et al., 1980 ) consisting of 59 items (Nauss, 1996 ). There are growing concerns regarding the content validity of this scale in current research because the qualitative data underlying the development of this scale (mid-1970s) has become less representative of effective leadership in the twenty-first century (Dobrotka, 2021 ). Additionally, the sample consisted exclusively of respondents from the 200 seminaries that were members of the Association of Theological Schools in the United States and Canada at that time. Consequently, the field could benefit from well-researched measures that can be used to evaluate religious leadership across different religious and cultural contexts. In order to develop such a scale, scholars should collaborate with practitioners to create evidence-informed scales that will benefit religious leaders and their communities as well. The work by McElroy et al. ( 2014 ), which describes the development of a measure of intellectual humility in the context of religious leadership, could be used for inspiration. Alternatively, researchers could build on and extend work in related fields, such as the development of the Authentic Leadership Inventory (Neider & Schriesheim, 2011 ), Spiritual Leadership Scale (Fry et al., 2005 ), Identity Leadership Inventory (Steffens et al., 2014 ), or Servant Leadership Survey (Van Dierendonck & Nuijten, 2011 ).

In short, the current scoping review provided an up-to-date overview of how religious leadership effectiveness has been assessed over the last 25 years in the academic literature. Although a focus on Christian leadership remained prevalent among the included references, recent references adopt a more dynamic and extensive approach to evaluating effective religious leadership by combining criteria from different target groups (i.e., leader, follower/member, and congregation) while using multiple measures and emphasizing the interactions between the leader and congregation.

Adams, A. W. (2013). Shepherd leadership of Church of God pastors and how this relates to ministerial effectiveness (Publication No. 3570373) [Doctoral dissertation, University of Phoenix]. ProQuest Dissertations Publishing.

Adeloye, G. O. (2011). Pastoral leadership: A sine-qua-non for effective church growth. Practical Theology: Baptist College of Theology, Lagos, 4 , 84–95.

Google Scholar  

Adetoyese, J. O. (2018). The marks of a successful pastor. The American Journal of Biblical Theology, 19 (7), 110–120.

Agee, B. R. (2001). Servant leadership as an effective approach to leadership in the church. Southwestern Journal of Theology, 43 (3), 7–19.

Al-Hilali, M. (2012). Transformational leadership and organizational effectiveness: A predictive study at American Muslim Organizations (Publication No. 3570201) [Doctoral dissertation, University of Phoenix]. ProQuest Dissertations Publishing.

Ammerman, N. T. (2020). Rethinking religion: Toward a practice approach. American Journal of Sociology, 126 (1), 6–51.

Article   Google Scholar  

Arksey, H., & O’Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8 (1), 19–32. https://doi.org/10.1080/1364557032000119616

Avolio, B. J., & Bass, B. M. (2004). Multifactor Leadership Questionnaire: Manual and sampler set (3rd ed.) Mind Garden.

Banfill, F. (2015). Multiple case studies in effective Africa leadership: A study of the leadership behaviors of effective local church pastors in the Africa inland church Tanzania Mara and Ukerewe diocese (Publication No. 10263321) [Doctoral dissertation, Indiana Institute of Technology]. ProQuest Dissertations Publishing.

Bass, B. M. (1985). Leadership and performance beyond expectations . Free Press.

Bass, B. M., & Riggio, R. E. (2005). Transformational leadership. Psychology Press. https://doi.org/10.4324/9781410617095

Bird, D. E. (1997). Personality factors as related to successful church planters among Texas Baptists (Publication No. 9720741) [Doctoral dissertation, Fuller Theological Seminary]. ProQuest Dissertations Publishing.

Boyatzis, R., Brizz, T., & Godwin, L. (2011). The effect of religious leaders’ emotional and social competencies on improving parish vibrancy. Journal of Leadership & Organizational Studies, 18 (2), 192–206. https://doi.org/10.1177/1548051810369676

Brauer, S. (2018). The surprising predictable decline of religion in the United States. Journal for the Scientific Study of Religion, 57 (4), 654–675. https://doi.org/10.1111/jssr.12551

Carter, J. C. (2009). Transformational leadership and pastoral leader effectiveness. Pastoral Psychology, 58 (3), 261–271.

Chaves, M., Roso, J., & Holleman, A. (2022). The National Survey of Religious Leaders: Background, methods, and lessons learned in the research process. Journal for the Scientific Study of Religion, 61 (3–4), 737–749. https://doi.org/10.1111/jssr.12803

Cloyd, M. (2019). Unseen success in Christian leadership failures. Journal of Biblical Perspectives in Leadership, 9 (1), 213–225.

ADS   Google Scholar  

Cobb-Hayes, J. (2018). An investigation of the relationship between authentic leadership and pastoral leadership effectiveness (Publication No. 13808904) [Doctoral dissertation, Indiana Wesleyan University]. ProQuest Dissertations Publishing.

Counts, J. D. (2020). Discovering leadership models that produce fruit within the Mid-Atlantic Episcopal district (Publication No. 28384349) [Doctoral dissertation, Ashland Theological Seminary]. ProQuest Dissertations Publishing.

Dadswell, D., & Ross, C. (2013). Church Growth Research Project: Church planting. Oxford Centre for Ecclesiology and Practical Theology Ripon College, Cuddesdon. https://www.churchofengland.org/sites/default/files/2019-06/cgrp_church_planting_from_cgrd_website.pdf

De Hart, J., & Van Houwelingen, P. (2018). Christenen in Nederland: Kerkelijke deelname en christelijke gelovigheid. Den Haag: Sociaal en Cultureel Planbureau. https://www.scp.nl/publicaties/publicaties/2018/12/19/christenen-in-nederland

Delaney, H., & Spoelstra, S. (2022). Transformational leadership: Secularized theology? In B. Caroll, J. Ford, & S. Taylor (Eds.), Leadership: Contemporary critical perspectives (3rd ed., pp. 75–93). SAGE.

DeNeal, D. A. (2019). Level 5 leadership and pastoral effectiveness: A study of lead pastors’ level 5 leadership score to church growth indicators (Publication No. 27670550) [Doctoral dissertation, Johnson University]. ProQuest Dissertations Publishing.

DeRue, D. S., Nahrgang, J. D., Wellman, N. E., & Humphrey, S. E. (2011). Trait and behavioral theories of leadership: An integration and meta-analytic test of their relative validity. Personnel Psychology, 64 (1), 7–52. https://doi.org/10.1111/j.1744-6570.2010.01201.x

Dinh, J. E., Lord, R. G., Gardner, W. L., Meuser, J. D., Liden, R. C., & Hu, J. (2014). Leadership theory and research in the new millennium: Current theoretical trends and changing perspectives. Leadership Quarterly, 25 (1), 36–62. https://doi.org/10.1016/j.leaqua.2013.11.005

Dobrotka, S. P. (2021). Measuring clergy effectiveness: The development of the Clergy Effectiveness Scale. In M. C. Bocarnea, B. E. Winston, & D. Dean (Eds.), Handbook of research on advancements in organizational data collection and measurements: Strategies for addressing attitudes, beliefs, and behaviors (pp. 293–313). IGI Global.

Chapter   Google Scholar  

Dodds, M. L. (1997). Case studies of pastoral leadership in the church, No. 19 (Publication No. 99724362) [Doctoral dissertation, Dallas Theological Seminary]. ProQuest Dissertations Publishing.

Douglas, S. M. (2015). Pastoral skills and qualities for effective ministry in Southern Baptist churches. Journal for Baptist Theology & Ministry, 12 (2), 31–43.

Dreibelbis, J. (2010). From maintaining to building communities of faith. Anglican Theological Review, 92 (1), 147–155.

Dye, W. H. (1998). Case studies of pastoral leadership in the church, No. 20 (Publication No. 9823645) [Doctoral dissertation, Dallas Theological Seminary]. ProQuest Dissertations Publishing.

Feaster, W. B. (2015). The relationship between time management behaviors and ministerial effectiveness among ministers of education in the state of Texas (Publication No. 3716280) [Doctoral dissertation, Dallas Baptist University]. ProQuest Dissertations Publishing.

Francis, L. J., Ratter, H., & Longden, G. (2015). Linking clergy psychological type profile and church growth in the Diocese of Chester: A pilot study. Journal of Beliefs & Values, 36 (3), 354–358. https://doi.org/10.1080/13617672.2015.1095544

Friedman, E. H. (2007). A failure of nerve: Leadership in the age of the quick fix. Church Publishing.

Friedman, E. H. (2011). Generation to generation: Family process in church and synagogue . Guilford Press.

Fry, L. W., Vitucci, S., & Cedillo, M. (2005). Spiritual leadership and army transformation: Theory, measurement, and establishing a baseline. Leadership Quarterly, 16 (5), 835–862. https://doi.org/10.1016/j.leaqua.2005.07.012

Garber, K. A. (2021). Developing a research-based approach to discovering characteristics and creating action steps that enhance pastoral effectiveness (Publication No. 28493783) [Doctoral dissertation, Regent University]. ProQuest Dissertations Publishing.

Garner, B. (2020). The resilient pastor: How to remain effective and finish well (Publication No. 28264457) [Doctoral dissertation, Biola University]. ProQuest Dissertations Publishing.

Giordan, G. (2014). Introduction: Pluralism as legitimization of diversity. In G. Giordan, E. Pace (Eds.), Religious pluralism . Springer, Cham. https://doi.org/10.1007/978-3-319-06623-3_1

Glatt, R. J. (2019). An examination of the correlation between church conflict and pastoral leadership effectiveness (Publication No. 27665046) [Doctoral dissertation, Trevecca Nazarene University]. ProQuest Dissertations Publishing.

Graham-Brown, F. B. (2020). Examining the pastoral servant leadership behaviors of metro New York City Baptist organizations and its impact on church effectiveness: A grounded theory study (Publication No. 28152784) [Doctoral dissertation, Sullivan University]. ProQuest Dissertations Publishing.

Green, C. C. (2001). Leadership theory of a Protestant minister. Journal of Leadership Studies, 8 (1), 77–95.

Gunardi, A., Setiadi, B. N., Hidajat, L. L., & Tjahjadi, S. P. L. (2022). Measurement of the effectiveness of Catholic priests’ pastoral services: A psychometric approach. Pastoral Psychology , 1–15. https://doi.org/10.1007/s11089-021-00985-y

Gyuroka, T. C. (2016). What do pastors in German-speaking Europe perceive as important leadership competencies in order to be effective pastoral leaders (Publication No. 10162853) [Doctoral dissertation, Andrews University]. ProQuest Dissertations Publishing.

Haskell, D. M., Flatt, K. N., & Burgoyne, S. (2016). Theology matters: Comparing the traits of growing and declining mainline Protestant church attendees and clergy. Review of Religious Research, 58 (4), 515–541. https://doi.org/10.1007/s13644-016-0255-4

Haslam, S. A., Reicher, S. D., & Platow, M. J. (2015). Leadership: Theory and practice. In M. Mikulincer & P. R. Shaver (Eds.), APA handbook of personality and social psychology, Vol. 2: Group processes (pp. 67–94) . American Psychological Association.

Haslam, S. A., Reicher, S. D., & Platow, M. J. (2020). The new psychology of leadership: Identity, influence and power (2 nd ed). Routledge. https://doi.org/10.4324/9781351108232

Hoffmann, J. P. (2013). Declining religious authority? Confidence in the leaders of religious organizations, 1973–2010. Review of Religious Research, 55 , 1–25. https://doi.org/10.1007/s13644-012-0090-1

Hudson, J. S. (2017). A competency model for church revitalization in Southern Baptist convention churches: A mixed-methods study [Doctoral dissertation, The Southern Baptist Theological Seminary]. Open Access Dissertation and Theses.

Hussey, I. (2013). Leadership in effective and growing Australian congregations: A study of three cases. Journal of Religious Leadership, 12 (1), 63–88.

Inglehart, R. F. (2020). Giving up on God: The global decline of religion. Foreign Affairs, 99 (5), 110–118.

Inglehart, R. F. (2021). Religion’s sudden decline: What’s causing it, and what comes next? Oxford University Press.

Book   Google Scholar  

Jacobsen, L. S. (2004). The leadership factor in church planting projects in Norway from 1990 to 2000 (Publication No. 3175621) [Doctoral dissertation, Regent University]. ProQuest Dissertations Publishing.

Jules, P. (2020). Description of leadership traits that contribute success within church or religiously affiliated organizations (Publication No. 28261923) [Doctoral dissertation, St. Thomas University]. ProQuest Dissertations Publishing.

Kane, M. N., & Jacobs, R. J. (2015). Perceptions of success and failure among aging Catholic priests. Journal of Religion, Spirituality & Aging , 27 (4), 288–304. https://doi.org/10.1080/15528030.2015.1046630

Kisling, R. A. (2007). Character for leadership: The role of personal characteristics in effective leadership behaviors (Publication No. 3292253) [Doctoral dissertation, Regent University]. ProQuest Dissertations Publishing.

Levac, D., Colquhoun, H., & O’Brien, K. K. (2010). Scoping studies: Advancing the methodology. Implementation Science, 5 (1), 1–9. https://doi.org/10.1186/1748-5908-5-69

Mahazan, A. M., & Abdullah, A. G. (2013). A model of imam’s leadership and mosque performance in Malaysia. Global Journal Al-Thaqafah, 3 (2), 53–64. https://doi.org/10.7187/GJAT432013.03.02

Main, J. M. (1998). Case studies of pastoral leadership in the church, No. 21 (Publication No. 9823648) [Doctoral dissertation, Dallas Theological Seminary]. ProQuest Dissertations Publishing.

Majovski, L. (1982). The Role of Psychological Assessment in Ministerial Selection [Unpublished doctoral dissertation, Fuller Theological Seminary].

Malony, H. N. (1984). Ministerial effectiveness: A review of recent research. Pastoral Psychology, 33 (2), 96–104. https://doi.org/10.1007/BF01086370

Malony, H. N. (2000). The psychological evaluation of religious professionals. Professional Psychology: Research and Practice , 31 (5), 521. https://doi.org/10.1037/0735-7028.31.5.521

Malony, H. N., & Hunt, R. A. (1991). Ministerial effectiveness. In H. N. Malony & R. A. Hunt (Eds.), The psychology of clergy (pp. 109–120). Morehouse.

Manners, A. T. (2008). Influence of transformational, autocratic, democratic, and laissez-faire leadership principles on the effectiveness of religious leaders (Publication No. 3370948) [Doctoral dissertation, University of Phoenix]. ProQuest Dissertations Publishing.

Marshall, J., & Olson, D. V. A. (2018). Is ‘spiritual but not religious’ a replacement for religion or just one step on the path between religion and non-religion? Review of Religious Research, 60 , 503–518. https://doi.org/10.1007/s13644-018-0342-9

Martinez, F. (2020). Best leadership practices of Protestant ministers in the Presbyterian Church of USA (Publication No. 28091689) [Doctoral dissertation, Pepperdine University]. ProQuest Dissertations Publishing.

McElroy, S. E., Rice, K. G., Davis, D. E., Hook, J. N., Hill, P. C., Worthington, E. L., Jr., & Van Tongeren, D. R. (2014). Intellectual humility: Scale development and theoretical elaborations in the context of religious leadership. Journal of Psychology and Theology, 42 (1), 19–30.

McKenna, R. B., & Eckard, K. (2009). Evaluating pastoral effectiveness: To measure or not to measure. Pastoral Psychology, 58 (3), 303–313. https://doi.org/10.1007/s11089-008-0191-5

Munn, Z., Peters, M. D. J., Stern, C., Tufanaru, C., McArthur, A., & Aromataris, E. (2018). Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Medical Research Methodology, 18 , 1–7. https://doi.org/10.1186/s12874-018-0611-x

Nauss, A. (1972). Problems in measuring ministerial effectiveness. Journal for the Scientific Study of Religion, 11 (2), 141–151. https://doi.org/10.2307/1384926

Nauss, A. (1994). Ministerial effectiveness in ten functions. Review of Religious Research , 36 (1), 58–69. https://doi.org/10.2307/3511652

Nauss, A. (1996). Assessing ministerial effectiveness: A review of measures and their use. Research in the Social Scientific Study of Religion, 7 , 221–251. https://doi.org/10.1163/9789004496231_014

Neider, L. L., & Schriesheim, C. A. (2011). The Authentic Leadership Inventory (ALI): Development and empirical tests. Leadership Quarterly, 22 (6), 1146–1164. https://doi.org/10.1016/j.leaqua.2011.09.008

O’Donnell, D. (1997). Can we predict successful priests? The Furrow, 48 (4), 229–234.

Oh, J., & Wang, J. (2020). Spiritual leadership: Current status and agenda for future research and practice. Journal of Management, Spirituality & Religion, 17 (3), 223–248. https://doi.org/10.1080/14766086.2020.1728568

Oney, R. M. (2009). Exploring the causal relationship of emotional intelligence to clergy leadership effectiveness (Publication No. 3392554) [Doctoral dissertation, Regent University]. ProQuest Dissertations Publishing.

Palmer-Atkins, C. B. (2018). An exploratory study of a nondenominational church and leadership behaviors, principle, strategies and practices (Publication No. 10684473) [Doctoral dissertation, Walden University]. ProQuest Dissertations Publishing.

Park, J. S. (2010). A study of effective lay small group leaders in Korean American churches: Fostering spiritual growth (Publication No. 3399094) [Doctoral dissertation, Trinity International University]. ProQuest Dissertations Publishing.

Park, Y. J. D. (2014). Women’s effective leadership in contemporary Taiwanese churches (Publication No. 3621907) [Doctoral dissertation, Biola University]. ProQuest Dissertations Publishing.

Parrott, R. (2003). Competency, spirituality, and core-identity in pastors. Ashland Theological Journal, 35 , 73–81.

Pearse, N. J. (2011). Effective strategic leadership: Balancing roles during church transitions. HTS Teologiese Studies / Theological Studies, 67 (2), 1–7. https://doi.org/10.4102/hts.v67i2.980

Pickens, D. M. (2000). Predicting ministerial effectiveness: A review of empirical research (Publication No. ED444072) [Doctoral dissertation, Biola University]. ERIC.

Powell, R., Ward, N., Bellamy, J., Sterland, S., Jacka, K., & Pepper, M. (2022). Enriching Church Life: A guide to results from National Church Life Surveys for local churches (3rd ed.). Sydney, NSW: NCLS Research.

Puls, T. R., Ludden, L. L., & Freemyer, J. (2014). Authentic leadership and its relationship to ministerial effectiveness. Journal of Applied Christian Leadership, 8 (1), 55–75.

Rafferty, A. E., & Griffin, M. A. (2004). Dimensions of transformational leadership: Conceptual and empirical extensions. Leadership Quarterly, 15 (3), 329–354. https://doi.org/10.1016/j.leaqua.2004.02.009

Rambo, L. R., & Haar Farris, M. S. (2012). Psychology of religion: Toward a multidisciplinary paradigm. Pastoral Psychology, 61 (5), 711–720. https://doi.org/10.1007/s11089-011-0372-5

Rotaru, L. G. (2019). Leaders that finish well: Characteristics and practices of successful ministry (Publication No. 22616661) [Doctoral dissertation, Asbury Theological Seminary]. ProQuest Dissertations Publishing.

Rowold, J. (2008). Effects of transactional and transformational leadership of pastors. Pastoral Psychology, 56 (4), 403–411. https://doi.org/10.1007/s11089-008-0121-6

Rumley, D. D. (2011). Perceptions of the senior pastors’ transformational leadership style and its relationship to the eight markers of natural church development (Publication No. 3477630) [Doctoral dissertation, Indiana Wesleyan University]. ProQuest Dissertations Publishing.

Scuderi, N. F. (2010). Servant leadership and transformational leadership in church organizations (Publication No. 3413541) [Doctoral dissertation, The George Washington University]. ProQuest Dissertations Publishing.

Small, K. J. (2011). Successful leadership in the early years of ministry: Reflections for leadership formation in theological education. Journal of Religious Leadership, 10 (1), 51–80.

Schwarz, C. (2006). Natural Church Development: A guide to eight essential qualities of healthy churches (7th ed.). St. Charles: Churchsmart Resources.

Schuller, D., Strommen, M., & Brekke, M. (1980). Ministry in America . San Francisco: Harper and Row.

Soldo, A. (2017). Examining the relationship between leadership style and leadership effectiveness among imams in U.S. mosques (Publication No. 10600624) [Doctoral dissertation, Colorado Technical University]. ProQuest Dissertations Publishing.

Spoelstra, S. (2017). Leadership and religion. In J. Storey, J. Hartley, J.-L. Denis, P. ‘t Hart, & D. Ulrich (Eds.), The Routledge companion to leadership (pp. 260–277). Routledge.

Starr, E. J. (2001). Simply complex: An examination of paradoxical leadership in faith-based organizations (Publication No. 3039623) [Doctoral dissertation, State University of New York at Albany]. ProQuest Dissertations Publishing.

Steen, P. W. (2008). Leadership of growing churches: The role of the senior pastor’s attributes and leadership practices (Publication No. 3350606) [Doctoral dissertation, Biola University]. ProQuest Dissertations Publishing.

Steffens, N. K., Haslam, S. A., Reicher, S. D., Platow, M. J., Fransen, K., Yang, J., Ryan, M. K., Jetten, J., Peters, K., & Boen, F. (2014). Leadership as social identity management: Introducing the Identity Leadership Inventory (ILI) to assess and validate a four-dimensional model. Leadership Quarterly, 25 (5), 1001–1024. https://doi.org/10.1016/j.leaqua.2014.05.002

Steffens, N. K., Munt, K. A., van Knippenberg, D., Platow, M. J., & Haslam, S. A. (2021). Advancing the social identity theory of leadership: A meta-analytic review of leader group prototypicality. Organizational Psychology Review, 11 (1), 35–72. https://doi.org/10.1177/2041386620962569

Strunk, J., Milacci, F., & Zabloski, J. (2017). The convergence of ministry, tenure, and efficacy: Beyond speculation toward a new theory of pastoral efficacy. Pastoral Psychology, 66 (4), 537–550. https://doi.org/10.1007/s11089-017-0760-6

Sundby, M., & Nienaber, S. (2009). The truth beneath the myths: What we can learn from a study of clergy competencies. Congregations, 36 (4), 18–21.

Swanson, H. P. (1999). Pastoral effectiveness: A study of differences among comparison groups of Seventh-day Adventist clergy (Publication No. 9937628) [Doctoral dissertation, Andrews University]. ProQuest Dissertations Publishing.

Taylor, J. T. (2019). A comparative analysis of leadership styles that identifies effective religious leaders (Publication No. 27744663) [Doctoral dissertation, Northcentral University]. ProQuest Dissertations Publishing.

Thiessen, J., Wong, A., McAlpine, B., & Walker, K. (2019). What is a flourishing congregation? Leader perceptions, definitions, and experiences. Review of Religious Research, 61 (1), 13–37. https://doi.org/10.1007/s13644-018-0356-3

Thornhill, J. (1999). Effective pastoral leadership: Forming disciples of the Lord. The Australasian Catholic Record, 76 (2), 144–151.

Timmins, F., Caldeira, S., Murphy, M., Pujol, N., Sheaf, G., Weathers, E., Whelan, J., & Flanagan, B. (2018). The role of the healthcare chaplain: A literature review. Journal of Health Care Chaplaincy, 24 (3), 87–106. https://doi.org/10.1080/08854726.2017.1338048

Article   PubMed   Google Scholar  

Tromp, P., Pless, A., & Houtman, D. (2020). ‘Believing without belonging’ in twenty European countries (1981–2008): De-institutionalization of Christianity or spiritualization of religion? Review of Religious Research, 62 (4), 509–531. https://doi.org/10.1007/s13644-020-00432-z

Van Dierendonck, D., & Nuijten, I. (2011). The Servant Leadership Survey: Development and validation of a multidimensional measure. Journal of Business and Psychology , 26 , 249–267. https://doi.org/10.1007/s10869-010-9194-1

Vardaman, D. L. (2013). Leading change: Exploring the relationship between transformational, transactional, and change-oriented leadership and their impact on leadership effectiveness among pastors in a Protestant denomination in the mid-western United States (Publication No. 3613541) [Doctoral dissertation, Indiana Wesleyan University]. ProQuest Dissertations Publishing.

Wasberg, G. D. (2013). Differentiation of self and leadership effectiveness in Christian clergy: A mixed methods study (Publication No. 3602885) [Doctoral dissertation, Capella University]. ProQuest Dissertations Publishing.

Washington, K. S. (2016). Spiritual leadership in religious organizations: A grounded theory study (Publication No. 10013592) [Doctoral dissertation, The University of the Rockies]. ProQuest Dissertations Publishing.

Watt, L., & Voas, D. (2015). Psychological types and self-assessed leadership skills of clergy in the Church of England. Mental Health, Religion & Culture, 18 (7), 544–555. https://doi.org/10.1080/13674676.2014.961250

Watt, W. M. (2014). Relational principles for effective church leadership. Journal of Leadership Education , 13 (2), 125–139. https://doi.org/10.12806/V13/I2/T1

Wilson, D. C. (2022). Defining leadership. Philosophy of Management , 1–30. https://doi.org/10.1007/s40926-022-00210-7

Wollschleger, J. (2018). Pastoral leadership and congregational vitality. Review of Religious Research, 60 (4), 575–585. https://doi.org/10.1007/s13644-018-0352-7

Yukl, G. (1999). An evaluation of conceptual weaknesses in transformational and charismatic leadership. Leadership Quarterly, 10 (2), 285–305. https://doi.org/10.1016/S1048-9843(99)00013-2

Yukl, G., & Gardner, W. (2020). Leadership in organizations (9 th ed.). Pearson.

Zhu, J., Song, L. J., Zhu, L., & Johnson, R. E. (2019). Visualizing the landscape and evolution of leadership research. Leadership Quarterly, 30 (2), 215–232. https://doi.org/10.1016/j.leaqua.2018.06.003

Download references

Acknowledgements

We thank Michiel van Elk and Stefan Paas for their valuable feedback and careful reading of the manuscript.

Author information

Authors and affiliations.

Faculty of Religion and Theology, Department of Beliefs and Practices, Vrije Universiteit Amsterdam, De Boelelaan 1105, 1081 HV, Amsterdam, The Netherlands

Annemarie Foppen

University of Humanistic Studies, Utrecht, The Netherlands

Joke W. van Saane

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Annemarie Foppen .

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Foppen, A., van Saane, J.W. Assessing Religious Leadership: A Scoping Review of Leadership Effectiveness Criteria and Current Trends in the Academic Literature. Pastoral Psychol 73 , 1–21 (2024). https://doi.org/10.1007/s11089-023-01113-8

Download citation

Accepted : 15 October 2023

Published : 15 November 2023

Issue Date : February 2024

DOI : https://doi.org/10.1007/s11089-023-01113-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Religious leadership
• Scoping review
• Effectiveness criteria
• Find a journal
• Publish with us
• Track your research