4b Questionable Research Practices

Week: 10/06

Learning Objectives

We will have a closer look at the scientific practice in psychology today. How sound or reliable is the psychological research today? What are the major problems and threads of good practice? Can we identify questionable research practices and publication biases? Are there any approaches to prevent these problems?

Required Reading
  • Wagenmakers, Wetzels, Borsboom, Maas, & Kievit (2012), An Agenda for Purely Confirmatory Research\(^\textbf{*}\)
  • Ioannidis, Munafò, Fusar-Poli, Nosek, & David (2014), Publication and other reporting biases in cognitive sciences\(^\textbf{**}\)
  • Wicherts (2017), The Weak Spots in Contemporary Science (and How to Fix Them)\(^\textbf{***}\)

\(^\textbf{*}\) Elaborates the difference between good and bad scientific practices.
\(^\textbf{**}\) Describes the problem of reporting biases in research and possible approaches to prevent these problems. The article talks predominately about “cognitive science”. The considerations are however valid for any disciple in psychology.
\(^\textbf{***}\) The article discusses the same issues and solutions as Ioannidis et al. (2014). What is new here are the questionable research practices HARKing and p-hacking. You can focus on that. Chapter 3 and 4 can be ignored anyway.

Tutorial Meeting

Question 1

What is the difference between exploratory and confirmatory research?

Question 2

What kind of research would Popper advise? Why?

Question 3

The three pictures in the graphic above stand for three concepts? What are the concepts and exam what it means?

Question 4

Often researchers simply do not have a clear-cut hypothesis. Does it mean that it is not “allowed” to perform significance test like a t-tests? What is your opinion?

Question 5

Some popular media talk in the last year about the so-called “replication crisis” in science and particular in psychology. This might heavily damage (or has damaged already) the reputation of our discipline.

What is the replication crisis? Why is psychology as science affected by this?

Question 6

A general psychological phenomenon is often mentioned in the context of the discussion of research practices and low replication rates is the confirmation bias.

What is that bias and how does it related to hypothesis testing and/or low replication rates?

Question 7

What are further potential reasons for the low replication rate in psychology?

Question 8

How should we change scientific practice to improve the quality and credibility of research output? Explain the approaches/methods and why they might be useful.

Study Checklist

  • Exploratory and confirmatory science
  • Replication rates / replication crises
  • Confirmation bias
  • Publication and reporting biases
    • selective outcome and analyses reporting
    • file-drawer problem
    • proteus phenomenon
  • Questionable research practices
    • reporting biases (see above)
    • HARKing
    • \(p\)-hacking
    • optional stopping (lecture 4)
  • Approaches to prevent biases
    • study registration or pre-registration
    • Transparency and Open Science (that is, the availability of data & analyses)
    • Replications
    • different publication practices & incentive structures