Skip to main content
It looks like you're using Internet Explorer 11 or older. This website works best with modern browsers such as the latest versions of Chrome, Firefox, Safari, and Edge. If you continue with this browser, you may see unexpected results.
Westminster College Logo

SSC 251 - Research Methods for Social Sciences: Using Scholarly Resources

Defining Key Terms

Operationalizes- In the social sciences, we try to measure data and ideas that are often qualitative, rather than quantitative. Operationalization means that you're defining what you're going to measure and how you will be measuring it. 

Reliability- A measure is reliable if it can be replicate and reproduced in a similar experiment. If you have reason to believe that You couldn't get the same results with this experiment and something was off- it probably isn't reliable. 

Validity- Will these results hold up outside of the experiment? When thinking about the validity of a measure, imagine how it would look in the real world. If the measure is too artificial, the results may not be valid.

CRAAP

(To be sung to any tune you’d like)

C is for currency, what’s the date?

R is for relevance, how does it rate?

A is for authority, what’s the author up to?

Another A for accuracy, is it all true?

One more thing, before you’re through-

P for purpose, is this meant to negate?

The CRAAP test is a general evaluation tool that can be applied to almost any source you find at college and in your life. To learn more about the CRAAP test and the specific list of questions it provides for evaluating sources, please see this guide from the California State University, Chico.

Special Evaluation Questions for Experiments

Operationalization:


What is the author trying to measure?

  • What are they using to measure?
  • How is the author measuring?
  • How are they accounting for other possible factors?

Reliability:

Is it reliable and replicable? 

  • If this experiment was conducted again, would the results be the same?
  • Could you replicate this experiment?

Validity:

Is the sample representative?

  • Who is included in the study?
  • If certain groups are left out, is that intentional?
  • How might a non-representative sample impact results?

Is the scenario/setting too artificial?

  • Is the experiment too dissimilar from real life?
  • Is it based on a "realistic situation"
  • Is the situation believable to the subjects?

Other questions to ask

  • Are there any possible alternative explanations for the results that the study overlooks? 
  • Are key terms carefully defined?
  • How do the researchers reduce and otherwise deal with possible demand characteristics?

Articles to Evaluate

Articles to CRAAP:

Cooke, B. J., & Farrington, D. P. (2016). The Effectiveness of Dog-Training Programs in Prison: A Systematic Review and Meta-Analysis of the Literature. The Prison Journal, 96(6), 854-876.

Nelson, J. (2017). 6 Things To Know About Prison Dog Training Programs. I Love Dogs. Retrieved from https://iheartdogs.com/6-things-to-know-about-prison-dog-training-programs/

Katcher, A., Beck, A. M., & Levine, D. M. (1989). Evaluation of a pet program in prison—the PAL project at Lorton. Anthrozoös, 2(3), 175-180.

CRAAP Test

Category

Awesome

Okay

Meh

Uh oh

Currency

  • This information has a published date (and maybe even an edited/revised date)

  • This was published within the last 10 years (or it’s a historical source)

  • This information has a published date.

  • This was published within my lifetime or it is older but it seems very important to research on this topic.

  • This information was published a long time ago and is probably out of date compared to research on my  topic.

  • You don’t know when this was written/published

Relevance

  • You will be able to use most of this information to help you analyze your question.

  • You will be able to use a decent chunk of this information, or this is a background source that is very relevant to your topic.

  • This is about a topic that is kinda similar to your topic. You might be able to use some of the background and introduction.

  • Upon closer examination, this is about something completely different from your topic

Authority

  • You can find the name of the author

  • The author is a known expert in this field (when you google their name, you can find other books/articles they’ve written on a similar topic.

  • The information has been through some type of review process

  • You can find the name of the author

  • The author is a known expert or they are writing directly about an experience they had.

  • You can find the name of the author.

  • This information is found on a .edu or .gov URL

  • You have no idea who the author is

  • This information is on a .com, .org, .net, .biz, etc. (and you didn’t find it through a library database)

Accuracy

  • This information seems accurate

  • The author cites other sources throughout

  • It seems like the author did original research or analyzed the original research of others

  • This information seems accurate.

  • The author respectfully acknowledges alternative viewpoints.

  • There are citations or links to other sites that seem to be of good quality.

  • The author links to a few other sources, but the quality of the other sources doesn’t seem great.

  • Some facts are vague and sketchy

  • There are no citations

  • There are multiple errors in spelling and grammar- like it wasn’t edited or revised

Purpose

  • The author makes their purpose clear in the introduction/early in the chapter

  • This is to add to a larger discussion happening in the field

  • The author acknowledges their own biases /the potential shortcomings of the work

  • The author doesn’t articulate their purpose

  • This was written to inform  students and the general population about what experts are researching

  • This was written to persuade you to believe in a certain point of view.

  • You don’t know who the author is

  • This was written to sell you something or to convince you that the author is right.