Principles of Source Evaluation

No matter what kind of source you are using, you can apply several basic principles to evaluate the source and determine whether and how you should use it. One common mnemonic device for these principles is CRAAP:

• Currency – Is the information up-to-date? For your topic, how recent does your information need to be?
• Relevance – Is the information truly applicable to your topic? Is it at an appropriate level (not too elementary or too technical) for what you need?
• Authority – What are the author’s credentials on the topic?
• Accuracy – Is the information supported by evidence? Has the information been reviewed or fact-checked? Is the information consistent with what you find in other sources?
• Purpose – What is the intention behind the source? What political, ideological, cultural, or personal biases are likely to have influenced the information being provided?

Although these basic principles can apply to any type of source, it can be very helpful to also understand the main types of sources you’re likely to encounter. First, knowing the type of source can serve as clues to your source evaluation. Scholarly sources, for example, are going to be considered more credible than non-scholarly sources. Second, knowing the type of source can give you some direction about specific red flags to look for when you’re evaluating that kind of source. For instance, when you know you have an article from a public affairs magazine, you’ll know to be extra conscious of political biases. Finally, different types of sources follow different documentation rules, and you’ll need to be familiar with the basic format for each of the most common types of sources. You can find out more about that in the chapter on Integrating Material from Sources.

A note about bias: The word “bias” has a negative connotation that I believe is a bit unfair. Many novice researchers believe that the goal is to find “unbiased” sources, but most experienced researchers will argue that such a thing is impossible. We all have beliefs, values, experiences, purposes, and limitations that make it impossible for us to provide a complete and unbiased perspective on a given topic, especially when we are examining complex issues. Even questions in the natural sciences, which may seem straightforward to answer objectively, typically depend on assumptions that may be widely accepted, but always remain open to revision as new evidence is discovered. Thus, instead of classifying sources as biased or not, I think it is more useful to consider what kinds of biases influence the source, how they influence the source, and how honest and clear the source is about those biases. Of course, this entire perspective is itself a demonstration of bias. My belief that bias is inevitable is certainly not shared by everyone! I’m just pretty skeptical of anyone who claims to have the one true perspective on any topic, and I think I’m right to be.

Understanding How Sources Vary by “Type”

The following section focuses solely on non-fiction sources, as you would not be using fiction for secondary research. It also focuses on the two most common categories within which you will be finding sources: books and periodicals. Of course, there are many other types of sources out there, but books and periodicals remain the most typical in academic work. Because library databases contain the full text of many books and periodical articles, you can often conduct your research entirely online without ever actually citing a website as a source.

Whether a source is considered scholarly or not is probably the most important distinction to note. Scholarly sources are the preferred sources for academic work, and you will often be required to use only scholarly sources or for a certain number or percentage of your sources to be scholarly. Both books and articles can be scholarly, but (annoyingly) how you determine whether a book is scholarly is different form how you determine whether an article is scholarly.

Scholarly sources are preferred, but there are credible non-scholarly sources out there, too. Be careful not to assume the terms “scholarly” and “credible” are equivalent; it’s better to think of scholarly sources as a very specific subset of the much larger pool of credible sources.

The remainder of this chapter explores the differences between books and periodical articles, how to determine whether each is considered scholarly, and how to evaluate the credibility of non-scholarly sources.

Books

With so many resources available online, defining exactly what counts as a “book” is more complicated than it seems like it ought to be. A basic definition is something like, “a work of substantial length that is not updated more often than once per year.” Though books may be republished in new editions, for most books, there is no set schedule for how often or when that will happen; it just depends on when updates to the text are needed. The only exceptions to that are reference books, some of which are updated annually. Another distinction is that each new issue of a periodical will contain all new content, whereas books retain most of their content from edition to edition, with just the relevant sections updated or re-organized as needed.

One great aspect of doing research through a library catalog and in subscription databases is that they will often label the type of source for you! If the catalog describes the work as a “Book” or “eBook,” you can feel pretty confident that’s what it is.

Books by Publication Category

Although the library will let you know that you indeed are looking at a book, you will still need to be able to identify the exact type of book you have. Different types of books have different degrees of credibility, and some types of books will have limitations on how you will be to use them.

When we divide nonfiction books into categories, we look at four variables:

• What type of publisher published the book?
  • If the book was published a university or other academic press, we’ll consider it “scholarly.” Note that you might have to do a little online sleuthing to find out details about the publisher!
• What is the intended purpose of the book?
• Who is the intended audience of the book?
• What are the credentials of the author?

Based on the publisher type, we will divide our non-fiction books into two broad categories: scholarly or non-fiction trade. We will then further divide the books based on their purpose, intended audience, and author credentials.

Scholarly books include three sub-categories: reference books, textbooks, and “other” scholarly books. In all three categories, the defining feature that makes it a scholarly book is that it is published by a university or other academic press. If the book is not published by a university or other academic press, it will automatically go into the “non-fiction trade” category.

The other features of scholarly books break down as follows:

• Reference Books
  • Purpose: Used for readers to look up specific pieces of factual information; not intended to be read straight through
  • Audience: May be general audience, as with a basic desk dictionary or general encyclopedia, or may be specific, as with a specialized work like Black’s Law Dictionary or the Diagnostic and Statistical Manual of Mental Disorders
  • Author: Expert
  • How to identify: Purely informational; often organized alphabetically by topic or in some other way that indicates that readers aren’t reading it straight through; title may contain words like dictionary, encyclopedia, atlas, or almanac
• Textbooks
  • Purpose: Used to teach students on a particular topic
  • Audience: Students
  • Author: Expert
  • How to identify: Often contains exercises and assignments; often addresses students directly (“you”)
• Other
  • Purpose: Presents an academic argument, theory, or research
  • Audience: Other experts
  • Author: Expert
  • How to identify: In-depth and highly technical; documents sources extensively; does not demonstrate characteristics of reference books or textbooks

All three of these categories are perfectly fine for you to use in academic work, but they aren’t all equally good, particularly when it comes to reference books or textbooks. If you are using reference books, you should try to use the more specialized kinds rather than those targeted at a general audience. If you use a textbook, the higher the level of student it’s for, the better; definitely avoid using textbooks for students below college level.

Non-Fiction Trade Books

Everything else will fall under the category of “non-fiction trade” (AKA “mass market”) books. Technically, publishers distinguish between those two terms, but they are similar enough for us to use interchangeably here. All of these books are intended to be sold for profit, hence the terms “trade” and “market.” The audience for non-fiction trade books tends to be fairly broad, especially as compared to scholarly books. Although most non-fiction trade books aren’t ideal to use, if you find one that is written by an expert, is targeted at an audience of educated adults, and doesn’t have something sketchy about its apparent purpose, it is probably acceptable to use for academic work, though still not as good as finding a scholarly source instead. If you feel like the book is trying to sell the audience something (whether that’s a product or an idea), make the author famous, or otherwise manipulate people in some way, definitely don’t use it!

Edited Collections: An Important Aside

One kind of book you may run into is what is called an “edited collection.” Edited collections contain multiple works written by various authors, all collected into one volume. Sometimes these works have been reprinted from other publications; if that’s the case, you’ll want to evaluate based on the original publication information, not based on the edited collection. (You will also need to cite these a bit differently from other types of books!)

Periodicals

Periodicals include magazines, newspapers, and scholarly journals. We call them periodicals because they are published periodically; new versions come out on a regular schedule, whether that’s daily, weekly, monthly, quarterly, or something else. Each new version is called an “issue,” and it typically contains all new content, distinct from previous issues. All of the issues from a given year are a “volume.” Thus, a periodical with quarterly issues will have four issues in a volume, while a periodical with weekly issues will have 52 issues in a volume.

Another feature that’s common across periodicals is that they collect content by different authors. Periodicals contain articles, which are different pieces written by different people, and then these pieces are reviewed by the editors of the periodical, arranged in some way, and then published. When you cite something from a periodical, you’re nearly always going to cite a specific article from a specific issue of a specific periodical. You’ll want to keep that in mind so you use the correct terminology and don’t do something like refer to the scholarly article you’re citing as a “journal.” Some journals have been publishing dozens of articles annually for well over a hundred years, so it’s important to understand that each article represents just one small slice of what a periodical has published!

When you’re identifying the specific type of periodical you have, it’s easier if you’re using the library catalog. Many catalogs and databases distinguish between newspapers, magazines, and scholarly journals in their listings. Some databases also have a specific label just for trade magazines, although sometimes they call them “trade journals” or “trade publications” instead.

Academic Journals

Academic journals, also called scholarly journals, are where scholars and researchers publish their work. These are where you find scholarly articles. Scholarly articles have been peer reviewed, which means they have been checked by a panel of experts on the subject and received their approval prior to publication. Most of the articles in an academic journal will have been peer reviewed.

Library catalogs and databases generally make it easy to tell when you have an article from an academic journal by including a label such as “peer-reviewed” or “academic journal” on the listing. However, although most of the contents of academic journals are peer-reviewed, many of these periodicals also publish editorials, books reviews, and letters to the editor, and the library catalog might label these articles as peer-reviewed incorrectly because of the journal they’re published in. In fact, these are not peer reviewed and should generally not be used for academic work. Luckily, these are typically clearly labeled with a descriptive word prior to the title, as in “Editorial: 2013 in Review” or “Book Review: The Rise of Corporate Feminism.” When you see a label like that, that’s your cue that that particular article is an exception and will not count as a scholarly source.

Understanding and Reading Scholarly Articles

Most people do not read scholarly articles in their entirety, at least not to begin with. Following are some tips and tricks to help you skim scholarly articles efficiently, so you know which you want to keep and which you want to toss out. Most scholarly articles open with an abstract. The abstract is a summary of the text, and by itself, may provide you enough information to categorize your article (or even to decide you don’t need or want it). Always start by reading the abstract!

We can further categorize scholarly articles into four common types: empirical research reports, interpretive analyses, literature reviews, and theoretical articles. These divisions can be blurry, but knowing what category your article most closely resembles will help you know how to efficiently approach it.

• Empirical Research Reports – these articles are most common in the natural and social sciences. They report the results of the authors’ own study. They are pretty consistently divided into Introduction, Methods, Results, and Discussion sections. As a reader, you can save yourself a lot of time by reading only  the Introduction and the Discussion sections. Start by reading the last paragraph of each, and then backtrack if you need more context. The Methods and Results are almost always unnecessary for anyone who is not aiming to replicate the research themselves.
• Interpretive Analyses – these articles are most common in law and the humanities. They argue for a particular interpretation of a text. Although they do not have a standard pattern of organization like empirical research reports do, you can generally apply a similar process. Read the introduction, skim the article to get a sense of how the sections are divided up (if they are at all), and then read the end. Figure out from there if you need to backtrack and read more.
• Literature Reviews – these articles are common across all disciplines, but most common in the natural and social sciences. They summarize, synthesize, and evaluate the research that has been done on a given topic, usually with a goal of identifying areas where more research is still needed. (You may also see that empirical research reports often contain a literature review section, which works to show how the study the authors conducted meets a need.) These can be a fantastic source of information to provide an overview of the current state of academic research on a particular topic.
• Theoretical Articles – these articles are common across all disciplines. They discuss and often make an argument about a theory or theories. They may be arguing that an accepted theory is not accurate, that a new theory is valuable, and/or proposing a new theory or synthesis of theories. You’ll probably want to skip these articles entirely, as they are mainly useful only for someone who is very advanced in a particular field.

You are likely to find some articles that don’t fit neatly into any of the above categories, and that’s fine, too. Remember that the principles of closed-form writing hold true. The introduction should let you know what to expect from the text, and from there, you can read even just the first sentence of each paragraph to construct a basic outline of its ideas.

Everything Else: Commercial (For-Profit) Periodicals

For periodicals, everything that isn’t from an academic journal will fit into one of the commercial periodical categories, as described below. Some of these types of sources are better than others, but regardless, for academic work, you should use articles from commercial periodicals sparingly.

Newspapers, News Magazines, and Public Affairs Magazines

Newspapers primarily contain relatively short articles about current events. They’re usually divided into sections, so you might find World News, U.S. news, local news, politics, business, et cetera, and they also contain opinion sections with editorials and columns. These might include things like advice columns, horoscopes, and movie reviews. Because the articles are often not in-depth, newspaper articles are not commonly used in academic work except potentially for background information.

News magazines and public affairs magazines are fairly similar to each other and can be difficult to distinguish between. News magazine articles are a bit longer than newspaper articles, but typically are not extremely long, and they usually come out a little less often than a newspaper does. Examples of news magazines would be Time, Newsweek, The Week, Christian Science Monitor, National Journal, U.S. News and World Report.

Public affairs magazines typically have much longer and more highly researched articles. They’re still typically about current social and political issues, often analyzing these topics with a particular political slant, and they also usually contain other features like book reviews, movie reviews, humor columns. Some of them also include fiction. Examples of these would include The New Yorker, National Review, Atlantic Monthly, Harper’s, Commonweal, and Congressional Quarterly. Because these works are often very in-depth, they are more commonly cited in academic work than newspaper or news magazine articles.

When you’re evaluating your news and public affairs sources, keep in mind that national and international publications generally have strong fact checking. They do not want to risk being sued, so their fact checking departments are primarily focused on correcting errors that might lead to lawsuits. It’s not as high a level of editorial review as you would find with a scholarly source, but it is still better than something that has no review at all.

You should also be aware of your source’s reputation and political bias. The website mediabiasfactcheck.com is pretty good to determine a source’s reputation and its typical bias. Keep in mind that sources can be biased without being untruthful, and a source having a clear bias doesn’t mean you can’t use it; it just means you have to be cautious about how you use it. If you find a source that you like, but you find that it’s strongly leaning left or right, try to find a source on the other side of the spectrum on the same topic. That’s going to improve your ethos by showing your lack of bias in your research, and it will probably expand your understanding of the issue as well. The table below provides an overview of some popular public affairs sources with their typical political leaning; keep in mind, though, that the political biases of publications change over time, vary across different authors, and are somewhat open to interpretation.

Trade Magazines

Trade magazines, also known as trade journals or trade publications, are written for people working in a particular profession or trade. These can be good resources, but it’s unlikely in most cases that you’ll run into these on your topics that you’re researching. Examples of these would include magazines like Advertising Age, Chemical and Engineering News, Architectural Digest. In some ways, they may seem similar to academic journals because they are so focused on a particular specialty, but the contents are not peer reviewed.

Niche Magazines

Usually niche magazines are written for general audiences with a particular interest, so these are the magazines you see on the stands at the store, Vogue, Car and Driver, People, et cetera. You should not use articles from niche magazines in academic work.