911������

Identifying Credible Sources of Health Information in Social Media: Principles and Attributes

 

 

 

Introduction

People seek, share, and receive health information from a wide variety of sources, such as health care professionals, insurance and pharmaceutical companies, family and friends, media, educational materials, advertisements, and the internet—including social media. Increasing numbers of Americans have turned to internet sources for health and medical information in recent years, with approximately three out of four searching for health information online today, and similar rates among Europeans [1,2]. However, both high- and low-quality health information can be found online, and few social media platforms (SMPs) [a] differentiate between credible and non-credible sources of information. Consequentially, consumers must make their own judgments about how much trust to place in a source and the quality of the information it shares. These judgments are influenced by their level of health and digital literacy, prior knowledge, personal situations, and personal beliefs [3].

“Misinformation” is that which conflicts with the best scientific evidence available at the time. “Disinformation” describes a “coordinated or deliberate” effort to spread misinformation in order to gain “money, power, or reputation” [1]. Social media allows both misinformation and disinformation to be disseminated much more rapidly and broadly than ever before [4]. The ability for people to tailor their preferences on SMPs to see information from only the sources they select raises concerns about “bubbles” or “echo chambers” that could reinforce existing beliefs (although recent research has challenged this notion [5]). However, consumers do not have to proactively seek information that confirms their beliefs; algorithms used by SMPs and other web platforms often recommend content on the basis of users’ past behaviors and expressed interests, leading to passive or incidental exposure [6]. In the case of low-quality health information, such reinforcement loops can be harmful.

The coronavirus disease 2019 (COVID-19) pandemic has demonstrated the potentially malign outcomes of this aspect of social media. Misinformation about the disease spread through social media and other online forums—often fueled by politicization of scientific information—has considerably harmed the adoption of recommended prevention and control behaviors and has decreased support for vital policies, such as vaccination [7]. Therefore, SMPs are capable of amplifying misinformation and disinformation in harmful ways, including those that may lead to poor outcomes for individual as well as population health [8]. The authors believe that these platforms have an important opportunity—and a growing responsibility—to intervene, not only to counteract these harmful trends but also to enhance consumers’ access and exposure to high-quality, science-based health information. Proactive interventions by SMPs are one potential approach, although not a sole solution, to the challenge of “platform governance,” an issue that has been the subject of increasing policy debate [9].

The tremendous reach of SMPs among broad and diverse audiences affords them unique potential to support health-promoting behaviors amid the COVID-19 pandemic, as well as other current and future health challenges. For example, the two current most popular SMPs used by organizations to share health information—Facebook and YouTube—reach 2.85 billion [10] and “over 2 billion,” [11] monthly active users, respectively [b]. This represents a significant portion of the world’s population, estimated by the U.S. Census Bureau to be nearly 7.8 billion people in June 2021 [12]. Harnessing the power of social media to elevate high-quality information could therefore have a truly transformative effect on health and well-being worldwide.

However, determining what constitutes high-quality health information is a complex and multidimensional process. Although SMPs are beginning to pilot strategies to elevate and label high-quality information, there are no public data available to demonstrate what works and no scientific or technical consensus about the most effective approach. Nevertheless, the urgency of ensuring access to high-quality health information necessitates action, even if such action is initially imperfect. The challenge will require collaboration among public and private actors to develop incremental and iterative solutions, with attention to transparency, accountability, and incorporation of feedback from a diverse set of stakeholders.

This paper focuses specifically on the evaluation of sources of health information, rather than content or design (discussed further under “Scope”). The authors offer initial principles and attributes for consideration by SMPs in their efforts to identify credible sources—with the ultimate goal of promoting access to high-quality health information. The guidance in this paper is therefore limited in scope and is offered as a starting point in what should be an ongoing process. This guidance will also need to be regularly revisited and updated according to changes in the online information ecosystem. SMPs should invest in ongoing, rigorous research and analysis of this subject; commit to transparency and continuous quality improvement; and build and sustain collaborations with scientific, health, ethical, and other communities to ensure an effective and accountable approach.

Although this paper is intended to inform the policies of SMPs, organizations that share health information through social media may find the principles and credibility attributes useful in assessing their own approach. Importantly, members of the public might also use this guidance to inform their personal evaluation of sources. Both groups should be engaged by SMPs and others seeking to improve the accessibility of high-quality health information in social media.

 

Background

In March 2021, the 911������ Academy of Medicine (NAM) launched a project to help identify principles for identifying credible sources of health information in social media, of which this paper is the principal output. Sponsored by YouTube’s Healthcare and Public Health Partnerships arm [c], the project was inspired by the goal of enhancing public access to evidence-based health information during the COVID-19 pandemic, although the issue has relevance beyond the current crisis.

The project involved an independent expert advisory group composed of multi-disciplinary experts in information governance, health information development, public health and health equity, social media and misinformation, and science communication (members of which also authored this paper), a public webinar, a public comment period, and other information-gathering activities. This paper does not constitute official recommendations from the NAM or the 911������ Academies of Sciences, Engineering, and Medicine (NASEM), nor does it represent an endorsement of any actions taken by YouTube or other SMPs following its publication.

 

Methods

 

Managing Conflict of Interest

The NAM is an organization whose influence stems in part from its reputation as a credible source of health information. Further, the NAM disseminates this information in part through social media [d]. To minimize conflict of interest (COI), the NAM took steps to ensure the independence and objectivity of the advisory group and this paper. This paper represents the opinions of the authors and does not reflect a consensus position of the NAM, NASEM, or the authors’ organizations. The authors did not receive payment from the NAM, NASEM, or YouTube for their contributions to this paper, and the authors’ declared individual COIs are included in this paper’s back matter. This paper has been revised in response to scientific peer review by individuals who were chosen for their expertise in social media, ethics, health literacy, law, communications, and policy but are unknown to the authors.

 

Deliberative Sessions

The authors met for four closed, deliberative sessions between March and June 2021. Representatives from YouTube attended the first 60 minutes of the initial session in order to explain the company’s current policies and future goals with regard to elevating high-quality health information and to answer questions from the authors. Representatives from YouTube did not attend any part of the subsequent deliberative sessions. Notes from all three sessions are available to the public on the project webpage: NAM.edu/AuthoritativeHealthSources.

 

Information-Gathering Public Webinar

On April 5, 2021, the NAM hosted a public webinar to gather information to inform the authors’ deliberations. The webinar was planned with the input of the authors, and all authors attended. The topics covered included background on YouTube’s goals with regard to elevating credible sources of health information; the health and social consequences of social media misinformation and disinformation; how health information is received at the community level; and unintended consequences of social media content moderation strategies. The session concluded with a question-and-answer session among the authors and presenters (see Box 1). The webinar was attended by approximately 400 members of the public. The webinar recording, transcript, slide presentations, and a written summary are available on the project webpage, along with a synthesis of questions and comments submitted by public attendees.

 

 

Preliminary Discussion Document and Public Comment Period

The authors created a four-page preliminary discussion document to solicit feedback on the project from interested parties, including researchers, providers of online health information, and members of the public. The document contained background on the project; preliminary definitions and source categories; and ethical, logistical, and public health considerations. The document was posted on the project webpage on April 5, 2021, where it remains available [13].

The NAM hosted a questionnaire to collect comments on the discussion document between 12:00 pm ET on April 5, 2021, and 11:59 pm ET on April 9, 2021 (see Appendix C). The comment opportunity was promoted via email to approximately 1,000 individuals who had registered to attend the webinar and/or signed up for the project mailing list, as well as shared through the NAM’s social media channels. In total, the NAM received 49 comments. Fourteen of the commenters provided feedback on behalf of an organization, while the remainder commented as individuals. Three commenters were from Canada, one was from Mexico, one was from Egypt, and the remainder were from the United States. The comments were analyzed, sorted into themes, and summarized by a contractor [e]; this synthesis is available on the project webpage and presented more briefly in Box 2. The authors reviewed all comments received and considered them in developing this paper.

 

Review of Existing Models for Evaluation of Source Credibility

The authors performed a scan of existing models for evaluating source credibility and/or information quality (see Box 3 and Appendix A). Major themes that emerge across these models include the importance of independence from profit motivations and bias; rigorous content review processes; transparency and accountability; and mission-driven policies.

 

 

Scope

Given the complexity of the task—including the volume of health information shared through social media and the controversial nature of evolving content moderation policies—the authors limit their guidance to what they believe is a feasible first step toward enhancing access to high-quality health information. Therefore, this paper focuses on the credibility of sources of health information, rather than the information shared by these sources. Source evaluation is a common means of curating a large volume of content because it eliminates the need to evaluate every piece of information individually (although advances in machine learning may soon increase the feasibility of large-scale content evaluation).

However, although a reasonable place to begin, source evaluation cannot yield a complete solution to the challenge of increasing access to high-quality health information in social media. The credibility of a source is, at most, an indicator of information quality and by no means a guarantee. Furthermore, even organizations with strong reputations for credibility are not infallible. For example, the World Health Organization (WHO) and the Centers for Disease Control and Prevention (CDC) were slow to acknowledge the role of airborne transmission in the COVID-19 pandemic, and the CDC recently removed three pieces of guidance related to the virus from its website for failing to adhere to the agency’s scientific standards [14, 15]. Therefore, the authors stress that identifying credible sources of information is a starting point only and must be supplemented by ongoing and iterative efforts to assess the quality of information.

The authors further limit their consideration in this paper to government and nonprofit organizations (including nonprofit news sources that share health information), not individuals (e.g., independent physicians with Facebook pages) or for-profit companies. Individuals require separate analysis because they lack the organizational infrastructure that is the basis of the authors’ approach to source evaluation in this paper. For-profits have a unique set of financial interests that, likewise, require a separate assessment.

However, the authors’ decision to omit consideration of individual and for-profit sources from this paper does not reflect a judgment of their credibility. Individuals and for-profits may be highly credible and are influential sources of health information in social media. Therefore, an assessment of their credibility, as well as the quality of the information they share, should be the focus of future analysis. It also should be noted that demarcations between individuals and organizations and between nonprofit and for-profit organizations are not always clear. Many of the principles laid out in this paper can apply to for-profits.

Finally, the authors have limited their present focus to organizations based in the United States, including those that provide information in languages other than English. While some of the guidance in this paper can be applicable to organizations outside the United States, additional research and the engagement of global partners will be needed for this work in the international context.

 

Key Terms

Terminological precision is critical to this task and was the subject of careful deliberation by the authors. The following are definitions and discussions of the key terms used in this paper.

 

Credible

For the purposes of this paper, the authors present their own definition of credible in the context of sources of online health information: “offering information that is consistent with the best scientific evidence available at the time and employing processes to reduce conflict of interest and promote transparency and accountability.” The principles that inform this definition are explained in the following section.

 

High-Quality Information

As noted in the Introduction, high-quality information is that which is “science-based” or consistent with the best scientific evidence available at the time. The state of science and knowledge is always evolving, so the marker of time is an important component of this definition. The evolution of knowledge is also the reason that more absolute terms, such as accurate, are less appropriate. Although this paper does not consider information quality directly, increasing access to high-quality information is the goal of the approach under discussion.

 

Health Information

The authors define health information as content pertaining to health conditions (physical and mental), behaviors affecting health, public health, population health, health care, health policy, or biomedical science.

 

Source

For the purposes of this paper, a source is an entity that offers health information through one or more social media channels branded to that entity. A channel is a proprietary forum where a source can share content (text, visual, video, or audio) and interact with social media users who choose to “follow” or “subscribe” to that channel, as well as users who discover the content through search engines or SMPs’ “recommended content” algorithms.

 

Credible Source of Health Information

Building on the definitions and discussion previously mentioned, the authors define credible source of health information as “a source that is likely to offer high-quality information and employ processes to reduce conflict of interest and promote transparency and accountability.” The use of the word “likely” in this definition reinforces the notion that source credibility does not necessarily equate to information quality, yet is still a useful indicator for consumers.

 

 

Foundational Principles

On the basis of their information gathering and deliberation, the authors developed the following foundational principles to guide identification of credible sources of health information in social media.

 

Principle 1: Science-Based

Sources should provide information that is consistent with the best scientific evidence available at the time and meet standards for the creation, review, and presentation of scientific content.

This principle reflects the authors’ conviction that scientific evidence is the only reliable predictor of health outcomes and therefore should be the foundation of health information provided to consumers. There are a number of attributes (e.g., use of citations) that help to indicate whether a source is sharing information that is consistent with the best scientific evidence available at the time, described in the following section.

 

Principle 2: Objective

Sources should take steps to reduce the influence of financial and other forms of conflict of interest or bias that might compromise or be perceived to compromise the quality of the information they provide.

This principle acknowledges that all sources have COIs or inherent biases. However, in order to be considered credible, sources should strive to separate the presentation of health information from profit motives and other biases (e.g., political). Sources should also disclose conflicts, as noted in the next principle.

 

Principle 3: Transparent and Accountable

Sources should disclose the limitations of the information they provide, as well as conflicts of interest, content errors, or procedural missteps.

The final principle acknowledges the fallibility of both organizations—which cannot eliminate COI and errors—and science itself. At the frontiers of understanding, scientific knowledge changes over time as more evidence becomes available and as existing evidence is analyzed in new ways. Scientific evidence, no matter how rigorous, can never guarantee a certain outcome for every individual or every context. Furthermore, Black, Indigenous, and People of Color (BIPOC) and other groups, such as LGBTQIA+ individuals and people with disabilities, are underrepresented within organizations traditionally considered authorities in science, meaning that the best available science might not fully reflect their experiences (discussed further in “Structural Bias”).

To maintain credibility, sources must clearly acknowledge the limitations of the information they share so that consumers can reach fully informed conclusions. Fundamentally, this last principle reflects one of the key themes among the public comments the authors received—the importance of protecting the right of individuals to autonomy and independent evaluation of the information they consume and the sources they choose to trust. It also acknowledges sources’ right to freedom of speech [f], but at the same time, requires sources to be fully transparent and provide all the context necessary for consumers to reach an informed judgment. However, protection of free speech and consumer autonomy must be balanced against the harms of misinformation and disinformation, as well as recent anti-science and “post-truth” trends in the media [16]. “Post-truth” refers to an environment in which scientific evidence is disregarded by some in favor of an alternative set of beliefs [17].

 

Credibility Attributes

Using the foundational principles as a scaffold, the authors identified a set of attributes that generally describe credible sources of health information (see Table 1). Not every source can display every attribute, but this should not preclude a general assessment of credibility. For example, a professional association may have a lobbying arm, which is counter to one of the attributes under the “objective” principle. However, the same organization might have a research arm that nearly or fully aligns with the attributes under the “science-based” principle. Furthermore, this organization may clearly disclose its lobbying activities to the public and maintain a strict firewall between political messages and health information for the public, thereby aligning with attributes under the “transparent and accountable” principle.

 

 

A credible source should demonstrate a preponderance of the attributes listed in Table 1 but should not be required to meet a formal numerical threshold. Although one of the key themes among the public comments that informed this paper was the desire for a simple rating system, the authors believe that such a calculus would be inappropriate given that each attribute is not necessarily of equal weight or importance. Instead, SMPs and consumers of health information could consider these principles and attributes as a framework to inform their own assessments of a source’s credibility. Further, sources of health information could consider using Table 1 as a roadmap to assess and potentially enhance their own credibility.

To avoid perfection paralysis, the authors believe that general alignment with the principles and attributes listed in Table 1, coupled with full disclosure of any deviations, could serve as a reliable initial signal of a source’s credibility. As noted in the sections that follow, some types of sources are subject to pre-existing, standardized vetting mechanisms that signal such alignment. However, there remain credibility concerns with these source types as a whole. All sources should publicly disclose deviations from the principles and attributes and be subject to other strategies to ensure information quality (described later in this paper).

 

Identifying Credible Sources of Health Information

 

Categorization

A very wide range of U.S. nonprofit or government sources provide health information through social media, including professional societies; health care organizations; public health departments; universities; think tanks; philanthropies; medical journals; grassroots community organizations; state, local, tribal, and territorial government health agencies, and more. The scope and size of these sources varies dramatically. Evaluating the credibility of each one individually, while a worthy eventual goal, is infeasible as a first step. Therefore, just as the evaluation of source credibility (versus information quality) is offered as an entry point into a larger task, the assessment of source categories (versus individual sources) is an initial tactic to assess credibility at scale.

 

Leveraging Pre-Existing, Standardized Vetting Mechanisms

Several categories of sources are subject to pre-existing, standardized vetting mechanisms that indicate general alignment with the authors’ foundational principles and attributes. Therefore, SMPs could consider sources in these categories as likely to be credible. However, these vetting mechanisms are not an exact match for the principles and attributes and reflect varying degrees of rigor. In addition, there are credibility concerns associated with each of these categories, as noted in the sections that follow. Therefore, sources in these categories should undergo additional vetting by SMPs (and also, perhaps, by independent third parties, as is discussed later), including some form of content review, as they refine their approaches to source evaluation.

 

Accredited Organizations

Accreditation is a voluntary process by which an organization earns a formal qualification as proof of its ability to meet quality and performance standards set by an accreditor. Although processes vary considerably by accreditor, accreditation is generally an intensive evaluation that involves extensive documentation and a site visit. Accreditation standards are set on the basis of research and evidence that demonstrates which qualities are associated with the highest degree of organizational effectiveness and the best possible outcomes for patients and other beneficiaries.

Because organizations must apply for reaccreditation on a regular basis (e.g., every few years), maintenance of accreditation indicates an ongoing commitment to transparency and accountability. Accreditors provide publicly accessible lists of organizations that have earned accreditation, as well as, in some cases, organizations that have failed to earn or maintain accreditation.

Accreditors themselves engage in continuous quality improvement and are held to a high standard of performance based on their recognition by federal authorities such as the Department of Education. In some cases, accreditors may be sponsored by relevant professional associations, which can suggest policy changes that ensure up-to-date accreditation criteria. For example, the Association of American Medical Colleges and the American Medical Association jointly sponsor the Liaison Committee on Medical Education, which accredits medical schools.

In another example, the Centers for Medicare & Medicaid Services (CMS) awards “deeming authority” to approved health care organization accreditors. Deeming authority means that accreditation can substitute for inspection by a state agency to determine whether a health care organization is eligible to receive payment from Medicare and Medicaid programs [18]. Similarly, the U.S. Department of Education provides a list of accreditors that are “recognized by the Secretary as reliable authorities concerning the quality of education or training offered by the institutions of higher education or higher education programs they accredit” [19].

Together, accredited organizations, accreditors, and collaborator organizations function as a network that supports consistent and high performance standards, continuous evaluation and improvement, and public transparency and accountability—although these characteristics are not specific to the context of sharing health information through social media.

Categories of accredited organizations that serve as sources of health information for the public include educational institutions (universities and health professions schools), health care organizations, health plans, and public health departments (see Box 5 for a summary and Appendix Table B-1 for a list of accreditors and what accreditation signifies for organizations in each category).

 

 

Credibility Concerns

As noted above, the accreditation mechanisms discussed in this section do not evaluate an organization’s credibility as a source of health information in social media. Rather, they affirm the general credibility of an organization in its role as a provider of a specific service, such as education or health care. Accredited organizations may not always adhere to the authors’ credibility principles and criteria (nonprofit health plans and ambulatory care centers, for example, maybe subject to strong profit motives). Furthermore, accreditation may not be an option for all sources of high-quality health information. Therefore, accreditation is an imperfect proxy for the evaluation of an organization’s credibility as a source of health information and should be viewed as a preliminary indicator.

Further, there is tremendous variation in rigor and scope across accreditation programs, so accreditation does not convey adherence to a common standard for credibility. It should also be noted that some accreditation programs offer accreditation to organizations that do not share high-quality health information as defined by the authors. For these reasons, accreditation cannot serve as a comprehensive evaluation of credibility as a source of health information and should be supplemented by other forms of vetting.

 

Academic Health and Medical Journals

Academic journals are scholarly periodicals that publish research or reports specific to a profession or field of study. Many journals promote their publications through social media; in addition, journalists summarize journal articles and share their topline findings through social media. Academic journals are generally affiliated with educational institutions or professional associations. Although they exercise editorial independence, many are owned by for-profit publishing corporations, such as Elsevier.

Academic journals generally adhere to rigorous processes to ensure scientific excellence and integrity, such as peer review. For example, health and medical journals indexed by the U.S. 911������ Library of Medicine (NLM) through MEDLINE must meet clear standards for “scope and coverage, editorial policies and processes”, scientific and methodological rigor, production and administration, and impact (see Box 3) [20]. A list of all MEDLINE-indexed journals is available from the NLM website [21].

Many journals also follow the Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals produced by the International Committee of Medical Journal Editors (ICMJE), an independent working group of medical journal editors [g]. Broadly, the recommendations establish standards for authorship, disclosure of financial and nonfinancial COI [h], editorial freedom, protection of research participants, corrections and retractions, and handling of scientific misconduct, among other guidance [22]. ICMJE publishes a list of journals that state that they follow the ICMJE recommendations on its website (ICMJE.org) but cautions that it cannot verify the accuracy or completeness of its list. However, echoing this paper’s position that the pursuit of perfection should not hamper incremental progress, ICMJE further notes that “maintenance of such a list may help to promote improvements in the quality of medical science and its reporting by indicating the standards many editors indicate they work to uphold” [23].

 

Credibility Concerns

Despite rigorous editorial and scientific review processes, journals sometimes allow the publication of articles that contain errors or misinformation, some of which may be retracted later. A social media user who consumes information from such articles may never learn of their retraction. Furthermore, errors or oversimplifications may be inadvertently introduced by entities who communicate about research published by a journal or other source—an issue that is of particular concern in social media, given the brief and ephemeral nature of social media content.

Second, as noted earlier in this paper, the state of science is always evolving. Many academic journals publish the results of unique experiments that have not been—and in some cases cannot be—reproduced by other scientists. This phenomenon is referred to as a lack of “reproducibility” (defined as consistent results using the same data and processes) or “replicability” (defined as consistent answers to the same scientific question, using different data and processes) [24]. However, the scientific community does not always view challenges in reproducibility or replicability as a problem. Rather, early hypotheses and approaches are built on by other scientists and become part of the evolution of knowledge. However, these nuances are not likely to be clear to the average consumer of health information, who may interpret the results of a single experiment as conclusive. Regardless of reproducibility and replicability, entities that consume and communicate about research may place too much weight on a single study that may prove to be an outlier in metaanalyses or literature reviews.

Finally, there are a growing number of journals that misleadingly claim to be high-quality health and medical journals and do not conform with publishing best practices, including those outlined in the previous section. These journals publish almost every submission they receive, charge authors for the publication of their articles, and do not use scientific peer review processes. Further, some are deliberately deceptive by using titles that are similar to established, reputable journals [25]. Prospective authors can perform their own research to determine whether a journal they are considering for publication of their articles is reputable. However, this step is not likely to be taken by someone who encounters information from a journal that seems reputable in social media. Furthermore, for the reasons outlined in this section, the simple inclusion of an article in MEDLINE does not guarantee the absence of errors or misinformation.

Together, these credibility concerns call for an aggressive and sustained effort to educate consumers of health information about the nature of scientific experimentation, quality assurance processes upheld by academic journals, and, generally, the factors by which they can judge source credibility and information quality. The need for such an effort is discussed further in the following section.

 

 

Government Organizations

Federal, state, and local government organizations operate under a number of provisions that support their credibility as sources of health information—primarily in the areas of transparency and accountability. As part of the system of checks and balances built into the U.S. government, the Constitution gives the legislative and judiciary branches oversight over the executive branch, where many federal organizations that serve as sources of health information reside (see Box 7). For example, Congress could compel the disclosure of documents or require the sworn testimony of representatives from the CDC to evaluate the agency’s response to COVID-19. Many federal agencies produce information at the request of Congress, with internal and external review to increase the quality of that information. For example, the U.S. Government Accountability Office produces nonpartisan fact-based information at the request of congressional committees and subcommittees and operates a hotline for the public to report on government fraud, waste, or abuse.

 

 

Public transparency laws reinforce this accountability. For example, the Freedom of Information Act requires disclosure, upon requests from the public, of documents controlled by the federal government (with specified exceptions), which allows for independent scrutiny and critique of government information. The Federal Funding Accountability and Transparency Act requires the disclosure of information about any organization that receives federal funds, while the Digital Accountability and Transparency Act sets standards for availability of this information on the federal website USASpending.gov. The Federal Advisory Committee Act requires that advisory committees to the federal government operate in an open and transparent manner so that the public can access and review the information these groups generate. Government agencies that receive a certain level of research funding (internal or external) must make the results of the research (such as in journal articles) available to the public for free. In addition, many external groups monitor information released by the government.

Similar provisions enhance the credibility of state and local governments as sources of health information, although such provisions may be less strict and may vary from one jurisdiction to another. Furthermore, state lawmakers can support highly divergent public health policies, as demonstrated by variations in mask-wearing guidance during the COVID-19 pandemic [26].

 

Credibility Concerns

Trust in the federal government is low among some groups, compromising its influence as a credible source of health information. According to a survey from the Pew Research Center, only 24 percent of Americans trust the federal government “to do what is right just about always or most of the time” [27] (this statistic refers to the federal government in general, not as a source of health information). Trends in trust of the government vary among political party affiliation as well as race and ethnicity, suggesting that perception of government credibility may vary across population groups [28].

Furthermore, provisions for transparency and accountability are important attributes of credible sources but are not a guarantee of high-quality information. Recent events have raised further concern, as in the example of the CDC’s removal of scientific guidance after allegations of undue political pressure on the agency’s staff [29]. Clearly, the degree of transparency and accountability upheld by the government at all levels is influenced by the leaders currently in office. Politicization can bias or limit the information that government organizations release. Government organizations may also withhold complete information because of privacy or security concerns, which may result in fragmented or distorted perceptions of issues. Therefore, although the authors believe that government organizations can generally be treated as credible sources, the principles and attributes identified in this paper should apply to them as well, and gaps in credibility should be further examined.

 

Nonprofit Organizations Not Subject to Standardized Vetting Mechanisms

Many categories of nonprofit organizations that are not subject to standardized vetting mechanisms serve as sources of health information. Some adhere to rigorous standards that align with the principles and attributes outlined in this paper, and some do not. There is no pre-existing, standardized mechanism for evaluating the credibility of sources in this category (although individual mechanisms exist). Therefore, SMPs that wish to assess the credibility of such sources should develop a standardized process for assessing alignment with the principles and attributes identified in this paper.

Table 2 lists types of organizations that share health information (excluding health care organizations, health plans, government organizations, and public health departments), along with the authors’ general observations about the credibility of organizations in each category, drawing from the principles and attributes.

 

 

Credibility Assessment Steps

To assess the credibility of sources not subject to preexisting, standardized vetting mechanisms that align with the authors’ principles and attributes, SMPs would need to collect and evaluate a standardized set of data. The means of data collection could be either primary or secondary (i.e., SMPs could undertake their own discovery process or rely on information provided by a source). For example, primary data collection might mean using technology to “crawl” a source’s website for evidence of citations, peer review processes, COI disclosures, etc. Secondary data collection might take the form of a credibility attributes and disclosures section that a source could provide to an SMP and post publicly on the homepage of its social media channel(s). This latter approach would require sources to self-regulate and comply with an informal “honor system” or “code of ethics.” For example, a source would have to decide whether the content of an advertisement posted alongside health information constitutes a conflict of interest that could compromise the quality of that information.

On the other hand, as previously mentioned, sources that are subject to such standardized vetting mechanisms can be afforded a preliminary assumption of credibility, as well as government organizations by virtue of their strict accountability practices. However, even sources in these groups should strive to display a preponderance of the authors’ credibility attributes and publicly disclose any deviations (as well as be subject to parallel content evaluation, as described in the following).

For any source type, SMPs’ approach to credibility assessment should include a human-led quality assurance (QA) program. Algorithms and other automated technologies are likely not capable of evaluating every nuance of the credibility attributes. The QA system should verify alignment with source credibility attributes as well as the quality of the information shared. To ensure that consumers are accessing high-quality health information, some form of content assessment is essential as a supplement to source assessment.

Although this paper does not provide guidance on principles or mechanisms for content assessment, the authors urge SMPs to invest in research and analysis to quickly build capacity in this area. Priority should be given to high-volume, highly influential sources of health information.

A particularly promising possibility for consideration by SMPs would be to outsource QA functions to an independent third party, either pre-existing or created for this purpose. This approach would bolster the objectivity and integrity of the process by reducing the role of SMPs, which are unavoidably conflicted by their financial and political stake in the performance of the system. To maximize independence and objectivity, this third party organization should not be solely funded by a single SMP.

 

Identifying Credible Sources for Consumers

Given the preliminary nature of the approach described in this paper and lack of insight into the policies and capabilities of SMPs, the authors do not offer an opinion as to whether or how SMPs should “elevate” credible sources of health information (e.g., via an algorithm). However, when it comes to options for identifying such sources for consumers, the authors suggest that SMPs do not at this time affix a formal label such as “credible” or “non-credible.” Instead, SMPs should provide contextual information that may serve as a “credibility cue” but stop short of a formal designation of credibility. For example, a platform could identify a source as an “Accredited Health Care Organization” and provide a link to a definition of that organization type, including general credibility observations such as those in Table 1. Of note, Twitter has taken a similar approach to identify government- and state-affiliated accounts on its platform [30]. This method would also align with calls to preserve the right of individuals to make independent judgments about the information they are consuming (although it should be acknowledged that consumers’ judgments are impacted by their degree of health and digital literacy, as described in the following).

Regardless of the specific approach to source identification, SMPs should be transparent with consumers about their policies as well as the principles and criteria that underlie them. Data collection and assessment, whether primary or secondary, as well as QA activities, should be performed at regular intervals (e.g., semiannually) to ensure sources’ ongoing alignment with the principles and attributes. Potentially, QA activities could also incorporate consumer feedback. Sources that do not demonstrate ongoing adherence should lose any public signal of credibility, and that loss should be made visible to consumers. SMPs may have or could develop lists of sources that are known proponents of harmful information, and these should be made public for the benefit of consumers.

SMPs should also monitor the policies of peer social media companies, both to be aware of how their own content may be repackaged on other platforms (i.e., perhaps stripped of important contextual information) and to learn, share, and reach alignment on approaches to common challenges.

 

 

Parallel Strategies to Supplement Source Assessment

SMPs’ approaches to source assessment should continue to be refined and improved, and, importantly, should be supplemented by other strategies (as called for in public comments; see Box 2).

 

Content Assessment

Once again, although a reasonable starting point, evaluation of source credibility alone is not an adequate tool to ensure social media users’ access to high-quality health information. SMPs must supplement source assessment strategies with an equally robust system for content evaluation. In the interim, SMPs should clearly explain the limitations of source credibility to consumers (i.e., a source deemed credible is likely to share high-quality information, but not guaranteed). Ultimately, source credibility and information quality should be integrated under a single, streamlined assessment system to maximize clarity and usability for both sources and consumers.

 

Management of Misinformation

The elevation of credible sources of health information, while an important contribution, is not enough on its own to counteract the harms of misinformation and disinformation. SMPs should maintain parallel strategies to address such false and inaccurate information, as well as sources that deliberately promulgate such information. Admittedly, management of misinformation is a highly complex challenge, both politically and legally, making the elevation of credible sources and high-quality information a potentially more feasible priority.

As noted previously, the state of science and knowledge is always evolving, and information that was once consistent with the best available evidence at the time can quickly become outdated. Credible sources can avoid the perception of misinformation by using clear date labels and striving to update content regularly.

 

Health Literacy, Culturally Competent Communication, and Community Relationships

Regardless of the eventual system for elevating credible sources and high-quality information, consumers will still make their own judgments about which sources and information to trust. In fact, one of the major themes from the public comment period, summarized in Box 2, was that SMPs must protect freedom of speech and the autonomy of users in accessing the information that they choose.

Users seeking health information may not be satisfied with an SMP’s assessment of source credibility or information quality. As explained by Lisa Fitzpatrick, founder and CEO of the Washington, DC-based community organization Grapevine Health, people are resourceful and often consult many sources, both online and offline, before reaching a conclusion [31]. Resourcefulness is an asset if people are empowered and provided with ready access to high-quality health information. Although many people have a high degree of health literacy, a large-scale effort is needed to ensure that resources are in place to support and educate all people to become savvy, informed, and science-literate users of social media. This concept is an important aspect of information equity [i] (across literacy levels, preferred languages, location/locality, etc.)—and, by extension, health equity.

Therefore, SMPs should invest in evidence-based health literacy and consumer education strategies to support the success of their in-house approaches to elevating credible sources and high-quality information. Such strategies could be designed and executed by the platforms themselves, but a better approach may be to delegate to independent third parties.

Consumers’ evaluation of the credibility of online information goes beyond source and content characteristics to considerations of design, or the way information is presented [3]. A source’s credibility is of little relevance if it fails to connect with its audience. As an illustration, Fitzpatrick shared a quote from a community member: “I don’t understand what doctors are saying, and if I don’t understand you, I can’t trust you” [31]. Several of the public comments the authors received echoed this point, noting that credible sources may not always present information in a manner that is appealing, engaging, or culturally competent.

“Cultural competence” refers to the ability to interact effectively with diverse audiences by recognizing and responding to variations in social, cultural, and linguistic needs [32]. Diversity should be considered across multiple dimensions, including but not limited to, race/ethnicity, education level, socioeconomic status, age, and political affiliation. Political affiliation may be particularly relevant to this discussion, as there is a 22 percent difference between Republicans who express a great deal of confidence in medical scientists (31 percent) and Democrats (53 percent) [33]. Republicans are also less approving of SMPs’ efforts with regard to content moderation, making it important to engage with bipartisan opinions in the design of policies such as those discussed in this paper [34].

Organizations that strive to be credible sources of health information should invest in strategies to improve their communication skills by using language and images that are informed by cultural contexts as well as understandable and engaging (at the same time, however, they must share complete and precise information and avoid oversimplification) [j]. They should also seek to build authentic, collaborative relationships at the community level. These efforts may help to foster consumers’ trust— which is all the more important as communication challenges during the COVID-19 pandemic may have impacted the perceived credibility of public health authorities in the United States [35].

As with efforts to advance health literacy and consumer education, SMPs should consider financial support for such endeavors to promote competent communication by credible sources on their platforms. Social scientists in the fields of science, risk, and health communication would make productive partners for SMPs in this work.

 

Source Self-Regulation and Consumer Evaluation

Rather than waiting for evaluation by SMPs, sources of health information that wish to be considered credible should take proactive steps to apply science-based, objective, and transparent and accountable principles to their institutional practices and presentation of information. The principles and attributes set forth in this paper may provide a useful starting point, as well as other resources (including those described in Box 3 and Appendix A). By making these efforts transparent and highly visible to the public and their peers, organizations can begin to build a self-sustaining, cultural “norm” for credibility that may ultimately increase the quantity of high-quality information in social media and have a cascading effect on the quality of information both online and offline.

A transparent set of principles used by sources to assert their own credibility would also support consumers’ independent judgments, as called for in the public comments that informed this paper. Consumer evaluation can supplement the efforts of SMPs to identify and elevate credible sources and high-quality information, as well as ensure accountability when sources fail to adhere to standards they set for themselves. However, there are several major limitations to the application of consumer evaluation in this context.

First, as noted previously, consumers have differing levels of health and digital literacy, which impacts their ability to assess the credibility of a source or the quality of a piece of information. This is why SMPs whose goal is to increase access to high-quality information should support efforts to advance health literacy. Second, consumer opinion is often captured through measurements of source or content popularity, comments, and recommendations, among other means [3]. Such elements can be easily “hijacked” by entities whose goal is to spread misinformation or disinformation, as has been seen in anti-vaccination disinformation campaigners’ use of “bots” and “trolls” to amplify their reach and engagement on Twitter [37]. Finally, as noted in Box 4, the interplay between consumer trust and credibility is complex and may be difficult to parse in consumer feedback. Nevertheless, consumer feedback is an important consideration, including in the design of source and content evaluation policies.

 

Ethical and Public Health Considerations

In an effort to ensure that the guidance in this paper increases access to high-quality health information, thereby promoting health, and minimizes harm, the authors have also outlined ethical and public health considerations for the approach described herein.

 

Ethical Considerations

Control of Information

Control of information emerged as a key concern during the public comment period. SMPs’ efforts to increase access to high-quality information (by elevating credible sources) may be perceived as censorship or an attempt to limit the autonomy of information consumers (see Box 2). Platforms should strive to engage consumers in the design and evaluation of such strategies and maintain public transparency around policies and actions taken. Policies should balance the need to minimize the harm that could occur through propagation of health misinformation (as seen during the COVID-19 pandemic [7]) with the right of the consumer to undertake a personal evaluation and judgment.

 

Structural Bias

BIPOC have historically been underrepresented in many institutions that enjoy a reputation for credibility today. For example, racial segregation persisted in U.S. universities until the latter half of the 20th century, and more insidious forms of prejudice endure to this day [37]. BIPOC researchers and their research insights are underrepresented in clinical and biomedical fields and are less likely to receive federal funding than their White counterparts [38]. According to 2020 data from the U.S. Office of Personnel Management, Black and other people of color made up 38 percent of the federal workforce but only 22 percent of Senior Executive Service positions [39]. This historic and ongoing underrepresentation of BIPOC and others in positions of influence in academia, science, health care, and government means information shared by these institutions may not always reflect the experiences of or resonate with these groups— thereby perpetuating underrepresentation and information inequity.

To counter this bias and increase equity and representation, SMPs should make a concerted effort to identify and promote sources that are not only credible but also trusted and utilized by diverse audiences, including BIPOC and other groups, such as new immigrants, LGBTQIA+ individuals, religious minorities, and people with disabilities. SMPs should assess consumer data to identify sources that are heavily utilized by marginalized groups and prioritize them for credibility evaluation and potential elevation. Partnerships with groups that represent the rights and health of such groups will be essential to the success of this effort.

 

Financial Conflict

A system that elevates credible sources of health information may create a new “credibility brand” that is profitable for both sources and SMPs themselves. As set forth in the authors’ principles and criteria, credible sources should take steps to ensure that financial and ideological interests do not compromise the presentation of science-based health information. However, financial gain and enhanced influence may be unavoidable collateral effects of designation as a credible source in social media channels.

SMPs should support research to understand the impact of credibility designations on the quality of information shared by sources, on sources’ level of influence both inside and outside social media, and on sources’ financial status. Advertisements should not be attached to high-quality health information shared through SMPs, both to minimize financial conflicts of interest and to avoid compromising the quality and accessibility of the information (e.g., with distracting and potentially misleading ads).

To uphold their integrity, SMPs should separate their own profit motives as much as possible from efforts to elevate credible sources of health information. One way to achieve this would be for platforms to work with independent third parties to design and implement source and content evaluation and moderation strategies.

 

Feasibility and Appropriateness of SMPs’ Role

This paper has made significant asks of SMPs beyond the initial task of identifying and elevating credible sources of health information. In addition to a quality assurance system for source evaluation, these include parallel strategies for content assessment and misinformation management, as well as collaborative efforts to promote equity and support public health research. Such activities will require a considerable investment of time and resources, and SMPs do not necessarily have a financial incentive to make this investment [41]. Some argue that platforms are actually disincentivized from interventions that could dampen profits driven by advertising and high rates of engagement with misinformation [42]. However, SMPs’ engagement in highly visible social responsibility efforts, as well as an enhanced corpus of credible sources that are trusted and relied on by consumers, may prove financially and politically beneficial.

The complexity of SMPs’ interests merits careful consideration of their role as moderators of health and other crucial public information. Although the authors believe that SMPs should take a proactive role across several dimensions, as outlined in this paper, government regulation and delegation to independent third parties should also be considered as potential supplementary approaches.

 

Public Health Considerations

 

Health Equity

A system to elevate credible sources—and thereby increase access to high-quality health information—must be designed to support health equity, as well as information equity, and not cement existing inequities. Diversity and inclusion are important components of a system that promotes health equity. As noted previously in the “Structural Bias” section, efforts to eliminate racial bias and foster diverse representation among credible sources of health information are important to avoid perpetuating health inequities.

Older people, adolescents, people with lower education and income levels, and racial and ethnic minority groups are more likely to face challenges related to limited health literacy [43]. Therefore, efforts to foster health literacy, engage in culturally competent communication, and build and sustain community relationships and trusted networks—as called for by the authors as a supplemental strategy to elevating credible sources—are supportive of health equity. SMPs should be sure to use accessible language when defining and explaining policies related to credible sources of information. SMPs should also consider digital literacy and strategies to address equity in access to high-quality digital information—a challenge referred to as the “digital divide.”

The digital divide is defined as “disparities in technology access and use [that have] compounding effects on existing inequities along income, educational, racial, and geographic dimensions” [44]. Although approximately three-quarters of Americans have access to high-speed broadband internet at home, rates vary significantly by education level and income. In 2019, only 46 percent of people with less than a high school education had broadband, compared with 93 percent of college graduates. In 2021, the rates were 57 percent for people making less than $30,000 annually and 92 percent for people making more than $75,000. Disparities by race and ethnicity are less dramatic but still significant: 80, 71, and 65 percent for white, Black, and Hispanic people, respectively, in 2021 [44]. Despite this, Black and Hispanic people are more deeply engaged in social media than whites across some dimensions [45].

The digital divide is an important consideration for SMPs as well as other platforms that facilitate the sharing of health information. If efforts to increase access to high-quality health information disproportionately benefit highly educated, wealthy, and white people, then they are cementing health and information inequities.

 

Contribution to Public Health Research

SMPs can be important partners in improving public health, but only if they agree to share data (e.g. backend data, algorithms and use engagement metrics, content moderation processes) with researchers. This paper provides guidance that is intended to increase access to high-quality health information and thereby promote individual and population health. However, SMPs alone have access to data that could form the basis of important health and behavioral research about how policies such as those discussed herein would actually affect the consumption of high-quality health information, as well as whether enhanced access to such information would favorably impact offline outcomes.

In addition to sharing such data as outlined previously, SMPs should be transparent about the methods they use to promote consumption of high-quality health information (e.g., through algorithmic recommendations), as well as the full scope of their policies and processes with regard to health information of any quality. As noted earlier, health misinformation and disinformation spread through social media can negatively impact health outcomes, and SMPs should take responsibility for and develop solutions to mitigate elements of their systems that enable such information to flourish.

SMPs’ reluctance or failure to share such data and moderation methods would prevent fully productive collaborations with the public health and behavioral science communities. To be considered credible themselves, platforms should make a public and highly visible commitment to transparency and accountability, especially with regard to data, policies and methods that could impact public health.

 

Conclusion

Increasing access to high-quality health information in social media is a complex challenge that requires navigating tremendous volume and variation among sources and information; the continuous evolution of science and knowledge; and significant ethical quandaries—chief among them, the need to protect free speech and consumers’ right to autonomy while minimizing the risk of harm from misinformation. To date, attempts at social media content moderation have been met with controversy and calls for federal regulation from both sides of the aisle [47]. Nevertheless, the potential influence of health information shared through social media on health outcomes, at both an individual and population level, compels action, even with the knowledge that such action will be incomplete at first.

This paper has presented guidance that could be leveraged by SMPs in identifying credible sources of health information—an incremental step toward the goal of enhancing access to high-quality health information. Although the scope of this discussion has been limited to U.S.-based nonprofit or government sources only, it is likely that many of the principles, attributes, and considerations can be applied to for-profit sources or individuals, as well as sources outside the United States. Efforts to fully assess the credibility of these sources, many of which are highly influential, should be an urgent priority for SMPs.

However, source evaluation is not a comprehensive solution. Several parallel strategies are required to ensure information quality and combat the risks of health misinformation, as detailed earlier. Foremost among these is a strategy to assess information quality and develop content moderation plans in response. The authors acknowledge the infeasibility of evaluating the accuracy and balance of every piece of health information on social media. However, a system of “spot checks” for quality and integrity, supported by machine learning technology but supplemented by expert human evaluation, is within reach. SMPs should invest in developing principles, guidelines, and applications for content assessment alongside strategies for source evaluation. Ultimately, the two approaches should be consolidated in a single system for the identification and elevation of high-quality health information. As previously noted, SMPs’ efforts in these areas should be supplemented by government regulation or delegation to independent third parties.

SMPs cannot, and should not, tackle this challenge alone. As those ultimately impacted by social media source or content curation strategies, consumers must be engaged in developing such strategies. Public engagement is also essential to promote transparency, foster trust, and minimize perceptions of censorship or paternalism. Organizations that use social media to share information have an important role as well, and should hold themselves publicly accountable to a set of principles that supports the quality of the information they share, as well as their own institutional credibility. Together, the actions taken by consumers, organizations, and SMPs can move toward greater availability and accessibility of high-quality health information.

Finally, consumers and organizations that utilize social media deserve to understand the mechanics and the outcomes of policies that affect the information they receive and share. Therefore, SMPs should make their source and content moderation practices (e.g., algorithms) and relevant data accessible to independent behavioral and public health researchers to analyze the effects on information consumption as well as offline behaviors. Without such information, consumers and organizations that collaborate with SMPs will have no way of knowing whether policies are justified or effective. To be effective partners in improving health, SMPs must make a firm commitment to transparency and accountability.

 

Notes

a] Social media platforms are for-profit companies that allow people and organizations to create profiles, interact with other users, share information, form groups or networks, and promote businesses or causes through various means.

[b] Facebook owns Instagram and Whatsapp (see https://about.facebook.com/company-info). YouTube is owned by Alphabet Inc., the parent company of Google (see https://abc.xyz).

[c] For an overview of the NAM project, see /programs/principles-for-defining-and-verifying-the-authority-of-online-providers-of-health-information. YouTube provided funding totaling $100,000 to offset the NAM’s operational expenses in facilitating the project. Karen DeSalvo, Chief Health Officer, Google Health, is an NAM member and serves on the NAM’s governing Council (YouTube is owned by Alphabet Inc., the parent company for Google). Garth Graham, Director and Global Head of Healthcare and Public Health Partnerships, is an NAM member (see https://blog.youtube/news-and-events/new-health-content-coming-youtube).

[d] The NAM has a presence on Facebook, Instagram, LinkedIn, Twitter, and YouTube.

[e] McCabe Message Partners, Washington, DC.

[f] It should be noted that freedom of speech has some limitations, including what is known as the “true threat” doctrine, which prohibits speech that constitutes a “clear and present danger,” such as the famous example of “shouting fire in a theater.” See https://fas.org/sgp/crs/misc/95-815.pdf.

[g] See http://www.icmje.org/about-icmje. The current members of the ICMJE are Annals of Internal Medicine, British Medical Journal, Bulletin of the World Health Organization, Deutsches Ärzteblatt (German Medical Journal), Ethiopian Journal of Health Sciences, JAMA (Journal of the American Medical Association), Journal of Korean Medical Science, New England Journal of Medicine, New Zealand Medical Journal, The Lancet, Revista Médica de Chile (Medical Journal of Chile), Ugeskrift for Laeger (Danish Medical Journal), the U.S. 911������ Library of Medicine, and the World Association of Medical Editors.

[h] Proper disclosure of conflicts of interest relies on the integrity of authors and cannot be fully enforced by journals.

[i] Information equity refers to equity of people’s access to information (e.g., through internet access) as well as the ability to understand and use that information to their benefit.

[j] For principles for making health information “understandable, useful, and navigable,” see /perspectives-2014-health-literacy-principles-guidance-for-making-information-understandable-useful-and-navigable.

 

Appendix A: Models for Assessment of Source Credibility

 

911������ Library of Medicine

The 911������ Library of Medicine (NLM) has developed at least three major source evaluation systems that provide useful examples for the task at hand: MEDLINE indexing, MedlinePlus indexing, and the Disaster Lit database.

 

MEDLINE Journal Selection

The 911������ Library of Medicine (NLM), the world’s largest medical library, uses stringent criteria to determine whether a journal should be included in MEDLINE, NLM’s premier bibliographic database. MEDLINE is the primary component of PubMed, a freely accessible online literature database developed and maintained by the NLM 911������ Center for Biotechnology Information (NCBI), with new citations added daily.

In considering whether a journal merits inclusion in MEDLINE, NLM considers “the scientific and editorial character and quality of a journal”. Several factors are used in making this decision, including the scientific policy set by the NLM Board of Regents [1], the suitability of the journal for the NLM Collection, and the recommendations of an NIH Federal Advisory Committee, the Literature Selection Technical Review Committee (LSTRC) [2].

The LSTRC evaluates journals for MEDLINE based on 5 critical elements, including scope and coverage, editorial processes, scientific rigor/methodological rigor, production and administration, and impact (more detail on all characteristics can be found at https://www.nlm.nih.gov/medline/medline_journal_selection.html [3]).

By using such clear and detailed guidelines for their journal selection process, NLM’s goal with MEDLINE is to demonstrate a high level of scientific rigor while reflecting sufficient geographic heterogeneity.

 

MEDLINEPlus

MedlinePlus is another resource created and curated by the NLM. Unlike MEDLINE, which is primarily used by researchers and professionals, MedlinePlus is a health information website for patients, their families and friends, and the general public. MedlinePlus aggregates health information from a variety of sources. “Some pages, like the medical encyclopedia and drug information, are licensed from outside sources. Other pages, like health topic pages, medical tests, and genetic pages, are created specifically for MedlinePlus” [4].

Information on MedlinePlus is reviewed and updated according to guidelines developed for each type of page. Health topics are “updated as new information becomes available, and broken links are checked and fixed daily” [4]. Medical tests are reviewed at least every 3 years, although content is also updated as needed between review cycles. Genetics pages are reviewed by experts in the field before being posted to MedlinePlus and with each substantial revision thereafter, and patient support and advocacy groups provide feedback on select content. MedlinePlus also adds new and updated articles and illustrations to the A.D.A.M. Medical Encyclopedia each month, and an A.D.A.M. Editorial Policy ensures that included content presents evidence-based health information.

In determining whether to link to a nongovernment website, MedlinePlus staff considers a range of criteria related to the website’s content, advertising, availability and maintenance, and user privacy. In examining a resource’s content, these criteria include:

• The website offers a description of the organization, including a mission statement that aligns with MedlinePlus’s aim to provide accurate health information to patients and their families and friends.
• “The organization provides accurate, science-based information that complements or enhances the health information found on MedlinePlus.
• The source of the content is trustworthy and transparent.
• The primary purpose of the website is educational, and the information is unbiased.
• The information provided is easy to understand, easy to navigate, and well organized.
• The website has original content.
• The website links only to reliable sources that meet MedlinePlus guidelines for links or other clearly stated guidelines established by the website” [5].

 

In addition to such content considerations, MedlinePlus gives preference to pages with no advertising. If the website has advertising, it must display an advertising policy that clearly separates educational content from advertising or sponsorship. MedlinePlus will not link to web resources that present content suggesting that Medline Plus endorses certain commercial products or services. For a resource to be linked on MedlinePlus, the website must also be consistently available, include contact information for customer support, and provide current information. Furthermore, websites must not require users to register, become a member of the organization, or pay a fee to view health information. Finally, MedlinePlus criteria specify that if a website collects personal information, it must clearly display “a privacy policy that explains how information collected from users remains private and confidential. If a website displays advertising, it should prevent advertisers and sponsors from collecting any personally identifiable information from users” [5].

In addition to these criteria for resources linked to MedlinePlus, for all the pages on MedlinePlus, a “page last reviewed” date is available near the bottom of the page to indicate “when the entire topic was reviewed and updated while a “page last updated” date indicates when any information was added to or removed from the health topic page” [5]. These additional indicators allow users to verify the currency of the content they are consuming.

 

The CRAP Test

Molly Beestrum, education and curriculum coordinator at Northwestern University’s Galter Health Sciences Library & Learning Center, developed a system known as the CRAP Test that can be used in deciding whether a website is a credible, valid source. The CRAP Test considers four major website attributes: currency, reliability, authority, and purpose. To apply Beestrum’s test, Colorado Community Colleges Online suggests asking the following questions:

“C�ܰ�����Գ���

• How recent is the information?
• How recently has the website been updated?
• Is it current enough for your topic?

Reliability

• What kind of information is included in the resource?
• Is content of the resource primarily opinion? Is it balanced?
• Does the creator provide references or sources for data or quotations?

Authority

• Who is the creator or author?
• What are the credentials? Can you find any information about the author’s background?
• Who is the publisher or sponsor?
• Are they reputable?
• What is the publisher’s interest (if any) in this information?
• Are there advertisements on the website? If so, are they clearly marked?

Purpose

• Is this fact or opinion? Does the author list sources or cite references?
• Is it biased? Does the author seem to be trying to push an agenda or particular side?
• Is the creator/author trying to sell you something? If so, is it clearly stated?” [6]

 

Clinical Practice Guidelines We Can Trust

The 2011 Institute of Medicine (IOM) study Clinical Practice Guidelines We Can Trust is another resource that may provide useful insights into the determination process for high-quality health information in social media [7]. Clinical practice guidelines (CPGs) help to reduce the level of uncertainty in clinical practice by establishing standards of care backed by strong scientific evidence. These standards “are informed by a systematic review of evidence and assessment of the benefits and costs of alternative care options” [7]. However, many different sets of CPGs have been developed and employed, to varying degrees of success. With nearly 27,000 guidelines in the 911������ Guideline Clearinghouse (NGC) and numerous additional commercial guidelines, it can be challenging to “identify guidelines based on high-quality development methods. Although the NGC provides a standardized summary of each CPG posting, describing its development methodology and evidence base and providing a link to the full guideline, the NGC makes no quality judgment” [7]. As a result, it can be difficult for stakeholders to be confident of CPG quality.

To combat this issue, Clinical Practice Guidelines We Can Trust proposed eight standards for developing trustworthy CPG and called for the development of a mechanism to identify guidelines that meet these standards. These standards include: “emphasizing transparency; management of conflict of interest; systematic review—guideline development intersection; establishing evidence foundations for and rating strength of guideline recommendations;
articulation of recommendations; external review; and updating” [7]. The report identifies three options in determining whether a CPG meets these standards: “1) identifying each guideline to see if it meets the specified standards; 2) certifying organizations producing guidelines that comply with quality standards; or 3) acknowledging standards compliance for each guideline production process prior to development of the guideline” [7].

Due to the large number of CPGs, the report suggests certification of organizations with trustworthy CPG development procedures rather than identification of individual trustworthy CPGs or identification of the development process for each CPG. This type of evaluation would entail “reviewing the procedures that applicant organizations use to produce various types of guidance, providing an identifiable mark to be placed on future CPGs of those organizations meeting accreditation requirements, and agreeing to maintain the approved processes during a 3-year accreditation period” [7]. This certification process “would not endorse particular drugs or treatment options for medical conditions or make clinical decisions about the guidelines it reviews” [7]. Instead, it would merely certify the organization’s guideline development process and identify the resulting CPGs as trustworthy.

 

Health on the Net Foundation Certification

The Health on the Net Foundation (HON) is an international nonprofit organization based in Switzerland. The HON Code of Conduct (HONcode) was created to help standardize the reliability of medical and health information available online by defining a set of rules to hold website developers accountable to basic ethical standards in the presentation of information. This voluntary certification system is based on an “active seal” concept that helps users identify sources of reliable information.

To be certified by HON, a website must formally apply for registration. If accepted, the site must comply with all eight principles enumerated in the HONcode:

1. “Authoritative: Any medical or health advice provided and hosted on this site will only be given by medically trained and qualified professionals unless a clear statement is made that a piece of advice offered is from a non-medically qualified individual or organization.
2. Complementarity: The information provided on this site is designed to support, not replace, the relationship that exists between a patient/site visitor and his/her existing physician.
3. Privacy: Confidentiality of data relating to individual patients and visitors to a medical/health Website, including their identity, is respected by this Website. The Website owners undertake to honor or exceed the legal requirements of medical/health information privacy that apply in the country and state where the Website and mirror sites are located.
4. Attribution: Where appropriate, information contained on this site will be supported by clear references to source data and, where possible, have specific HTML links to those data. The date when a clinical page was last modified will be clearly displayed (e.g., at the bottom of the page).
5. Justifiability: Any claims relating to the benefits/performance of a specific treatment, commercial product, or service will be supported by appropriate, balanced evidence in the manner outlined previously in Principle 4.
6. Transparency: The designers of this Website will seek to provide information in the clearest possible manner and provide contact addresses for visitors that seek further information or support. The webmaster will display his/her E-mail address clearly throughout the Website.
7. Financial disclosure: Support for this Website will be clearly identified, including the identities of commercial and non-commercial organizations that have contributed funding, services, or material for the site.
8. Advertising policy: If advertising is a source of funding, it will be clearly stated. A brief description of the advertising policy adopted by the Website owners will be displayed on the site. Advertising and other promotional material will be presented to viewers in a manner and context that facilitates differentiation between it and the original material created by the institution operating the site” [8].

 

To determine whether a site adheres to these standards, sites requesting certification complete an interactive, online questionnaire that tells them what they must add or modify to conform to the HONcode principles. Then an HON team member inspects the site to verify compliance. Once a site has been verified, it identifies itself with the blue and red HONcode hyperlink (or “active”) seal displayed in a prominent location, usually at the bottom of the homepage. Subscribing sites are subject to unannounced checks by HON to ensure continued compliance, and HON also relies on user reports to maintain website reliability.

 

URAC Health Content Provider and Health Website Certification

URAC provides health care organizations with evidence-based accreditation programs, including health content provider and health website certification. URAC accreditation involves a five-phase, voluntary process that requires reaccreditation every 3 years and is designed to facilitate continuous quality improvement. In the first phase, applicants provide URAC with standard information about their organization as well as specific information related to the type of certification being sought. In the second phase, a lead reviewer evaluates the submitted documents to determine whether they comply with URAC’s standards. During this step, the lead reviewer may provide recommendations to the applicant on how to revise an application to conform to URAC’s policies. The third phase involves an on-site validation review to ensure that the organization is following the standards in practice. In the fourth step, the URAC review team presents an anonymous report to a voluntary accreditation committee. This team of health experts is familiar with URAC standards and determines whether an organization receives full or partial accreditation. The final phase of the accreditation process includes ongoing monitoring such as random surveillance and required reporting of quality measures.

In all certification programs, URAC employs these five phases to focus on risk management, operations infrastructure, performance monitoring and improvement, and consumer protection and empowerment. Certification-specific standards are then designed to further support these goals. For instance, the” Health Content Provider accreditation process examines key areas such as disclosures, health content and service delivery, quality oversight committee, policies and procedures, health content and personal health management, and accountability” [9]. In contrast, considerations that factor into earning a Health Website Accreditation include privacy and security, health content editorial processes, disclosure of financial relationships, website linking policies, and consumer complaint processes.

 

Appendix A References

1. 911������ Library of Medicine. 2021. Board of Regents. Available at: https://www.nlm.nih.gov/od/bor/bor.html (accessed July 2, 2021).
2. 911������ Library of Medicine. 2021. 911������ the Literature Selection Technical Review Committee (LSTRC). Available at: https://www.nlm.nih.gov/medline/medline_about_lstrc.html (accessed July 2, 2021).
3. MEDLINE. 2021. Journal Selection for MEDLINE. Available at: https://www.nlm.nih.gov/medline/medline_journal_selection.html (accessed June 30, 2021).
4. U.S. 911������ Library of Medicine. 2021. Collection Development Guidelines of the 911������ Library of Medicine. Available at: https://www.ncbi.nlm.nih.gov/books/NBK518693/pdf/Bookshelf_NBK518693.pdf? (accessed June 23, 2021).
5. U.S. 911������ Library of Medicine. 2021. Guidelines for Links. MedlinePlus. Available at: https://medlineplus.gov/about/using/criteria/ (accessed June 23, 2021).
6. CCCOnline Library. 2021. Evaluating Sources Toolkit. Available at: https://ccconline.libguides.com/c.php?g=242130&p=2185475 (accessed June 16, 2021).
7. Institute of Medicine. 2011. Clinical Practice Guidelines We Can Trust. Washington, DC: The 911������ Academies Press. https://doi.org/10.17226/13058.
8. Team HON. 2019. Discover the 8 Principles of the HONcode in 35 Languages. Health on the Net. Available at: https://www.hon.ch/cgi-bin/HONcode/principles.pl?English (accessed June 16, 2021).
9. URAC. 2021. Health Content Provider Certification. Available at: https://www.urac.org/accreditation-cert/health-content-provider-accreditation/ (accessed June 23, 2021).

 

 

Appendix B

TABLE B-1: Types, Definitions, and Accreditors of Nonprofit Accredited Organizations That May Serve as Sources of Health Information in the United States and What Accreditation Signifies

 

 

 

Appendix C

 

Questionnaire Used to Collect Public Comments on the “Preliminary Discussion Document” for the 911������ Academy of Medicine Project on “Principles for Defining & Verifying the Authority of Online Providers of Health Information”

 

Background

In order to enhance the accessibility of trustworthy health information on its platform, YouTube asked the 911������ Academy of Medicine (NAM) to identify preliminary definitions of “authoritative” sources of health information and the criteria by which these sources derive and maintain their authority. The NAM project will also outline ethical and public health considerations for large-scale content curation strategies.

An expert advisory group will gather information and deliberate in order to author a peer-reviewed discussion paper for publication in NAM Perspectives in summer 2021. Papers published in NAM Perspectives are individually authored and do not reflect consensus positions of the NAM, the 911������ Academies, or the authors’ organizations. See project webpage >>

YouTube has informed the NAM that outcomes from this project will help to inform YouTube’s identification and raising of “authoritative” sources of health information, but will not disadvantage sources that do not meet the requirements of specific authoritative source categories. Further, the NAM understands that YouTube will identify and characterize such “authoritative” sources in order to provide context for users but will not confer a formal designation of authority or trustworthiness at the source level. The project will also generate principles that may be of use to online platforms other than YouTube.

 

Public Discussion Document

To inform the project, the NAM is seeking feedback on a public discussion document containing preliminary principles and questions. The public comment period will last from 12 pm ET on April 5, 2021, until 11:59 pm ET on April 9, 2021.

An anonymized synthesis of feedback received through this process will be posted on the project webpage after the comment period closes. The NAM reserves the right to disregard feedback it considers to be off topic or inappropriate.

Download the discussion document here >>

To leave a comment, please complete the form below.

First and last name of commenter:*

Email address of commenter:*

If you are commenting on behalf of an organization, please enter the organization name here:

One-sentence summary of your comments:*

Complete comments:*

Would you like to receive email updates about this project?*

[ ] Yes
[ ] No

* = Question is required

 

---

Join the conversation! 

  Authors of an #NAMPerspectives discussion paper call for engagement between social media platforms, consumers, & health organizations to determine what makes high-quality health information more available and accessible. Read proposed approaches:

  High-quality health information must be understandable, engaging, & culturally competent. In order to increase access & availability, those involved need to work hard to gain the trust of people at the community level: #NAMPerspectives

  “The potential for information shared through social media to influence health outcomes necessitates action by social media platforms to enhance access & exposure to high-quality, science-based information.” Read more: #NAMPerspectives

 

Download the graphics below and share them on social media!

 

 

 

 

 

References

1. Swire-Thompson, B., and D. Lazer. 2019. Public Health and Online Misinformation: Challenges and Recommendations. Annual Review of Public Health 41:433-451. https://doi.org/10.1146/annurevpublhealth-040119-094127.
2. Zhao, Y., and J. Zhang. 2017. Consumer Health Information Seeking in Social Media: A Literature Review. Health Information & Libraries Journal 34:268-283. https://doi.org/10.1111/hir.12192.
3. Sun, Y., Y. Zhang, J. Gwizdka, and C. B. Trace. 2019. Consumer Evaluation of the Quality of Online Health Information: Systematic Literature Review of Relevant Criteria and Indicators. Journal of Medical Internet Research (21)15:e12522. https://doi.org/10.2196/12522.
4. Chou, W.-Y. S., A. Gaysynsky, and J. N. Cappella. 2020. Where to Go from Here: Health Misinformation on Social Media. American Journal of Public Health (110)S3:S273-S275. https://doi.org/10.2105/AJPH.2020.305905.
5. Guess, A. M. 2021. (Almost) Everything in Moderation: New Evidence on Americans’ Online Media Diets. American Journal of Political Science. https://doi.org/10.1111/ajps.12589.
6. Chou, W.-Y. S., A. Gaysynsky, and R. C. Vanderpool. 2021. The COVID-19 Misinfodemic: Moving Beyond Fact Checking. Health Education & Behavior (48)1:9-13. https://doi.org/10.1177/1090198120980675.
7. Lee, J. J., K.-A. Kang, M. P. Wang, S. Z. Zhao, J. Y. H. Wong, S. O’Connor, S. C. Yang, and S. Shin. 2020. Associations Between COVID-19 Misinformation Exposure and Belief With COVID-19 Knowledge and Preventive Behaviors: Cross-Sectional Online Study. Journal of Medical Internet Research (22)11:e22205. https://doi.org/10.2196/22205.
8. Nyhan, B. 2021. The Problem of Social Media Misinformation and How to Address It (Webinar Presentation, 911������ Academy of Medicine, April 5, 2021). Available at: /event/defining-the-authority-of-online-providers-of-health-information (accessed June 16, 2021).
9. Popiel, P., and Y. Sang. 2021. Platforms’ Governance: Analyzing Digital Platforms’ Policy Preferences. Global Perspectives (2)1:19094. https://doi.org/10.1525/gp.2021.19094.
10. Facebook. 2021. Facebook Reports First Quarter 2021 Results. Available at: https://investor.fb.com/investor-news/press-release-details/2021/Facebook-Reports-First-Quarter-2021-Results/default.aspx (accessed June 16, 2021).
11. YouTube. 2021. YouTube for Press. Available at: https://www.youtube.com/intl/en-GB/about/press (accessed June 16, 2021).
12. U.S. Census Bureau. 2021. U.S. and World Population Clock. Available at: https://www.census.gov/popclock/world (accessed June 16, 2021).
13. NAM (911������ Academy of Medicine). 2021. Preliminary Discussion Document for Public Comment. Available at: /wp-content/uploads/2021/04/Public-Discussion-Document-NAM-Project-on-Principles-for-Defining-Verifying-the-Authority-of-Online-Providers-of-Health-Information.pdf (accessed June 16, 2021).
14. Tufekci, Z. 2021. Why Did It Take So Long to Accept the Facts 911������ Covid? The New York Times. Available at: https://www.nytimes.com/2021/05/07/opinion/coronavirus-airborne-transmission.html (accessed June 16, 2021).
15. Centers for Disease Control and Prevention. 2021. Agency Guidance Review. Available at: https://www.cdc.gov/coronavirus/2019-ncov/downloads/communication/Guidance-Review.pdf (accessed June 16, 2021).
16. Hotez, P. J. 2020. Combating Antiscience: Are We Preparing for the 2020s?. PLoS Biology (18)3:e3000683. https://doi.org/10.1371/journal.pbio.3000683.
17. McIntyre, L. C. 2018. Post-truth. Cambridge, MA: MIT Press.
18. The Joint Commission. 2018. Federal Deemed Status and State Recognition. Available at: https://www.jointcommission.org/-/media/tjc/idev-imports/topics-assets/facts-about-federal-deemed-status-and-state-recognition/federal_deemed_status_12_12_181pdf.pdf (accessed June 16, 2021).
19. U.S. Department of Education. 2021. Accreditation in the United States. Available at: https://www2.ed.gov/admins/finaid/accred/accreditation_pg6.html#RegionalInstitutional (accessed June 16, 2021).
20. NLM (911������ Library of Medicine). 2021. Journal Selection for MEDLINE. Available at: https://www.nlm.nih.gov/medline/medline_journal_selection.html (accessed June 16, 2021).
21. NLM. 2021. List of All Journals Cited in PubMed®. Available at: https://www.nlm.nih.gov/bsd/serfile_addedinfo.html (accessed June 16, 2021).
22. ICMJE (International Committee of Medical Journal Editors). 2019. Recommendations. Available at: http://www.icmje.org/recommendations (accessed June 16, 2021).
23. ICMJE. 2021. Journals Stating That They Follow the ICMJE Recommendations. Available at: http://www.icmje.org/journals-following-the-icmje-recommendations (accessed June 16, 2021).
24. NASEM (911������ Academies of Sciences, Engineering, and Medicine). 2019. Reproducibility and applicability in science. Washington, DC: The 911������ Academies Press. https://doi.org/10.17226/25303.
25. ICMJE. 2021. Responsibilities in the Submission and Peer Review Process. Available at: http://www.icmje.org/recommendations/browse/roles-and-responsibilities/responsibilities-in-the-submission-and-peer-peview-process.html (accessed June 16, 2021).
26. Gostin, L. O., I. G. Cohen, and J. P. Koplan JP. 2020. Universal Masking in the United States: The Role of Mandates, Health Education, and the CDC. JAMA (324)9:837-838. https://doi.org/10.1001/jama.2020.15271.
27. Pew Research Center. 2021. Americans See Broad Responsibilities for Government; Little Change Since 2019. Available at: https://www.pewresearch.org/politics/2021/05/17/americans-see-broad-responsibilities-for-government-little-change-since-2019 (accessed June 16, 2021).
28. Pew Research Center. 2021. Public Trust in Government: 1958 – 2001. Available at: https://www.pewresearch.org/politics/2021/05/17/public-trust-in-government-1958-2021 (accessed June 16, 2021).
29. Sun, L. H. 2021. CDC Identifies Public-Health Guidance from the Trump Administration That Downplayed Pandemic Severity. Washington Post. Available at: https://www.washingtonpost.com/health/2021/03/15/cdc-removes-some-trump-era-guidance (accessed June 16, 2021).
30. Twitter. 2021. 911������ Government and State-Affiliated Media Account Labels on Twitter. Available at: https://help.twitter.com/en/rules-and-policies/state-affiliated (accessed June 16, 2021).
31. Fitzpatrick, L. 2021. Credible Messengers in the Community: Lessons Learned in a Pandemic (Webinar Presentation, 911������ Academy of Medicine, April 5, 2021). Available at: /event/defining-the-authority-of-online-providers-of-health-information (accessed June 16, 2021).
32. Agency for Healthcare Research & Quality. 2019. Cultural Competence and Patient Safety. Available at: https://psnet.ahrq.gov/perspective/cultural-competence-and-patient-safety#_ednref4 (accessed June 16, 2021).
33. Funk, C., B. Kennedy, and C. Johnson. 2020. Trust in Medical Scientists Has Grown in U.S., But Mainly Among Democrats. Pew Research Center. Available at: https://www.pewresearch.org/science/2020/05/21/trust-in-medical-scientists-has-grown-in-u-s-but-mainly-among-democrats (accessed June 16, 2021).
34. Vogels, E. A., A. Perrin, and M. Anderson. 2020. Most Americans Think Social Media Sites Censor Political Viewpoints. Pew Research Center. Available at: https://www.pewresearch.org/internet/2020/08/19/most-americans-think-social-media-sites-censor-political-viewpoints/#views-about-whether-social-media-companies-should-label-posts-on-their-platforms-as-inaccurate-are-sharply-divided-along-political-lines (accessed June 16, 2021).
35. Tufekci, Z. 2021. 5 Pandemic Mistakes We Keep Repeating. The Atlantic. Available at: https://www.theatlantic.com/ideas/archive/2021/02/how-public-health-messaging-backfired/618147 (accessed June 16, 2021).
36. Ortiz-Sánchez, E., A. Velando-Soriano, L. Pradas-Hernández, K. Vargas-Román, J. L. Gómez-Urquiza, G. A. Cañadas-De la Fuente, and L. Albendín-García. 2020. Analysis of the Anti-Vaccine Movement in Social Networks: A Systematic Review. International Journal of Environmental Research and Public Health (17)15:5394. https://doi.org/10.3390/ijerph17155394.
37. Harris, L. M. 2015. The Long, Ugly History of Racism at American Universities. Available at: https://newrepublic.com/article/121382/forgotten-racist-past-american-universities (accessed June 16, 2021).
38. Clark, U. S., and Y. L. Hurd. 2020. Addressing Racism and Disparities in the Biomedical Sciences. Nature Human Behaviour 4:774-777. https://doi.org/10.1038/s41562-020-0917-7.
39. Partnership for Public Service. 2020. A Revealing Look at Racial Diversity in the Federal Government. Available at: https://ourpublicservice.org/blog/a-revealing-look-at-racial-diversity-in-the-federal-government (accessed June 16, 2021).
40. Tufekci, Z. 2021. Unintended Consequences of Large-Scale Content Curation Strategies (Webinar Presentation, 911������ Academy of Medicine, April 5, 2021). Available at: /event/defining-the-authority-of-online-providers-of-health-information (accessed June 16, 2021).
41. Oates, S. 2020. The Easy Weaponization of Social Media: Why Profit Has Trumped Security for U.S. Companies. Digital War 117-122. https://doi.org/10.1057/s42984-020-00012-z.
42. Muvuka, B., R. M. Combs, S. D. Ayangeakaa, N. M. Ali, M. L. Wendel, and T. Jackson. 2020. Health Literacy in African-American Communities: Barriers and Strategies. Health Literacy Research and Practice (4)3:e138-e143. https://doi.org/10.3928/24748307-20200617-01.
43. NASEM. 2020. Using Research and Technology to Address Compounding Disparities: Proceedings of a Workshop–in Brief. Washington, DC: The 911������ Academies Press. https://doi.org/10.17226/25738.
44. Pew Research Center. 2021. Internet/Broadband Fact Sheet. Available at: https://www.pewresearch.org/internet/fact-sheet/internet-broadband (accessed June 16, 2021).
45. Auxier, B. 2020. Social Media Continue to Be Important Political Outlets for Black Americans. Pew Research Center. Available at: https://www.pewresearch.org/fact-tank/2020/12/11/social-media-continue-to-be-important-political-outlets-for-black-americans (accessed June 16, 2021).
46. Kern, R. 2021. Tech Targeted on Antitrust, Liability After Facebook’s Trump Ban. Bloomberg Government. Available at: https://about.bgov.com/news/tech-targeted-on-antitrust-liability-after-facebooks-trump-ban (accessed June 16, 2021).