The Black Goat – Details, episodes & analysis

In 2012, Rink Hoekstra received two emails on the same day. One was from a journal editor, telling him that a manuscript was being rejected based on the recommendations of two reviewers. The other was from one of those reviewers, complimenting the paper and congratulating him on a job well done. The reviewer, Fiona Fidler, discovered that her review had been altered, and Rink and Fiona teamed out to figure out why. We spoke with Rink in 2018 about what happened, but we held on to the interview in anticipation of the episode being covered by the press. There's now an article out in Science, by journalist Cathleen O'Grady. In our conversation we talk about what happened, and we broaden out to a discussion of publication ethics. Why would an editor want to change a review without asking the reviewer? How does that damage a system that already has so little accountability? And what can authors or reviewers do when they suspect something is up?

Link:

• Delete offensive language? Change recommendations? Some editors say it’s OK to alter peer reviews, by Cathleen O'Grady

The Black Goat is hosted by Sanjay Srivastava, Alexa Tullett, and Simine Vazire. Find us on the web at www.theblackgoatpodcast.com, on Twitter at @blackgoatpod, on Facebook at facebook.com/blackgoatpod/, and on instagram at @blackgoatpod. You can email us at letters@theblackgoatpodcast.com. You can subscribe to us on iTunes or Stitcher.

Our theme music is Peak Beak by Doctor Turtle, available on freemusicarchive.org under a Creative Commons noncommercial attribution license. Our logo was created by Jude Weaver.

This is episode 86. Our interview was recorded on October 26, 2018; the introduction was recorded on October 28, 2020.

Academics are under enormous stress right now, raising the possibility of a rising rate of burnout. Longtime structural trends in higher education have increased pressures for demonstrable productivity. On top of that are a global pandemic, resistance and backlash to calls for racial justice, and unstable politics in the U.S., the U.K., and elsewhere. In this episode we discuss burnout in academia. We focus on an emerging perspective from the healthcare field that describes burnout as resulting from moral injury. How is this idea relevant to people working in academia? In what ways can we be hurt by being trapped between the ideals and values that brought us into the field and the demands of our working environments? What can we do about it? Plus: A letter about reviewing papers from the global south that do not fit into the usual discourse.

Links:

• Andrew Wilson's Twitter thread about burnout
• Moral Injury and Burnout in Medicine: A Year of Lessons Learned by Wendy Dean and Simon Talbot, Stat
• Burnout Is About Your Workplace, Not Your People by Jennifer Moss, Harvard Business Review
• Job Burnout by Christina Maslach, Wilmar Schaufeli, and Michael Leiter, Annual Review of Psychology

The Black Goat is hosted by Sanjay Srivastava, Alexa Tullett, and Simine Vazire. Find us on the web at www.theblackgoatpodcast.com, on Twitter at @blackgoatpod, on Facebook at facebook.com/blackgoatpod/, and on instagram at @blackgoatpod. You can email us at letters@theblackgoatpodcast.com. You can subscribe to us on iTunes or Stitcher.

Our theme music is Peak Beak by Doctor Turtle, available on freemusicarchive.org under a Creative Commons noncommercial attribution license. Our logo was created by Jude Weaver.

This is episode 85. It was recorded on September 16/17 (US/AUS), 2020.

Many important questions about cause and effect are impractical to answer with a randomized experiment. What should we do instead? In this episode we talk about doing causal inference with observational data. Has psychology's historical obsession with internal validity led it, ironically, to think about causal inference in an unsophisticated way? Can formal analytic tools like directed acyclic graphs (DAGs) tell us how to do better studies? Or is their main lesson don't bother trying? How do norms and incentives in publishing help or hurt in doing better causal inference? Plus: We answer a letter about applying to psychology grad school when your background is in data science.

Links:

• Thinking Clearly About Correlations and Causation: Graphical Causal Models for Observational Data, by Julia M. Rohrer
• That one weird third variable problem nobody ever mentions: Conditioning on a collider, by Julia Rohrer
• The selection-distortion effect: How selection changes correlations in surprising ways, by Sanjay Srivastava

The Black Goat is hosted by Sanjay Srivastava, Alexa Tullett, and Simine Vazire. Find us on the web at www.theblackgoatpodcast.com, on Twitter at @blackgoatpod, on Facebook at facebook.com/blackgoatpod/, and on instagram at @blackgoatpod. You can email us at letters@theblackgoatpodcast.com. You can subscribe to us on iTunes or Stitcher.

Our theme music is Peak Beak by Doctor Turtle, available on freemusicarchive.org under a Creative Commons noncommercial attribution license. Our logo was created by Jude Weaver.

This is episode 76. It was recorded on March 16, 2020.

The path from theory to study consists of a thousand decisions, big and small. How and how much do these decisions matter? We discuss a recent crowdsourced meta-study that tried to find out. Fifteen teams of researchers were given 5 different hypotheses and told to design a study to test them, then they ran all the studies and got widely varying results. What are the implications of this study for how we should think about the role of theory in study design? What does it say about the different functions of direct and conceptual replications? Is this evidence of hidden moderators? How predictable were the differences in results, and were they predictable because of differences in the study designers' expertise, biases, or something else? Plus: We answer a letter about getting scooped on a systematic review.

Links:

• Crowdsourcing hypothesis tests: Making transparent how design choices shape research results by Landy et al. in Psychological Bulletin
• 200 researchers, 5 hypotheses, no consistent answers. Coverage at Wired by Christie Aschwanden

The Black Goat is hosted by Sanjay Srivastava, Alexa Tullett, and Simine Vazire. Find us on the web at www.theblackgoatpodcast.com, on Twitter at @blackgoatpod, on Facebook at facebook.com/blackgoatpod/, and on instagram at @blackgoatpod. You can email us at letters@theblackgoatpodcast.com. You can subscribe to us on iTunes or Stitcher.

Our theme music is Peak Beak by Doctor Turtle, available on freemusicarchive.org under a Creative Commons noncommercial attribution license. Our logo was created by Jude Weaver.

This is episode 75. It was recorded on January 30, 2020.

How important is expertise in conducting replications? Many Labs 4 was a large, multi-lab effort that brought together 21 labs, running 2,220 subjects, to study that question. The goal was to compare replications with and without the involvement of the original authors to see if that made a difference. But things got complicated when the effect they chose to study - the mortality salience effect that is a cornerstone of terror management theory - could not be replicated by anyone. In this episode, we talk about the implications of Many Labs 4. What does and doesn't this study say about the importance of expertise and "secret sauce" in conducting replications? Should we necessarily expect that the effect of expertise is to make effects larger or more likely to replicate? What does this replication mean for terror management theory, which has been the focus of hundreds of studies and a 2010 meta-analysis that concluded that mortality salience effects are real and substantial? What place does terror management theory hold in the history of social psychology, and how is that similar or different from ego depletion, another theory that was supported by a meta-analysis but not by multi-lab replications? Plus: we answer a letter about weaponizing tenure if you get it.

Links:

• Many Labs 4: Failure to Replicate Mortality Salience Effect With and Without Original Author Involvement
• A blog post about ML4 by three of its authors
• Two Decades of Terror Management Theory: A Meta-Analysis of Mortality Salience Research, published in 2010

The Black Goat is hosted by Sanjay Srivastava, Alexa Tullett, and Simine Vazire. Find us on the web at www.theblackgoatpodcast.com, on Twitter at @blackgoatpod, on Facebook at facebook.com/blackgoatpod/, and on instagram at @blackgoatpod. You can email us at letters@theblackgoatpodcast.com. You can subscribe to us on iTunes or Stitcher.

Our theme music is Peak Beak by Doctor Turtle, available on freemusicarchive.org under a Creative Commons noncommercial attribution license. Our logo was created by Jude Weaver.

This is episode 74. It was recorded on January 22, 2020.

The Graduate Record Exam - the GRE - is widely used in graduate school admissions. In recent years however, a number of graduate programs, including a few in psychology, have stopped requiring it in a movement that has been dubbed "GRExit." In this episode we discuss the arguments around using the GRE in graduate admissions. What is the evidence for and against its validity? For and against the presence of bias against various groups? How much do we know about validity and bias in the other materials routinely considered in admission, like grades, undergraduate institution, research experience, and letters of recommendation? Are arguments over the GRE just a proxy for larger and more difficult arguments about the purpose and social value of graduate education? And for programs that are dropping the GRE, what are they doing instead, and how will we know what the effects of that are? Plus: we answer a letter about giving authorship to undergrads who made minimal contributions to a project in order to help them get into grad school. 

Links:

• CRediT - Contributor Roles Taxonomy
• A comprehensive meta-analysis of the predictive validity of the graduate record examinations: Implications for graduate student selection and performance, by Kuncel, Hezlett, and Ones (2001)
• Standardized tests predict graduate students' success, by Kuncel and Hezlett (2007)
• Beyond the GRE: Rethinking Admissions Procedures and GRE Scores Are Poor Predictors

The Black Goat is hosted by Sanjay Srivastava, Alexa Tullett, and Simine Vazire. Find us on the web at www.theblackgoatpodcast.com, on Twitter at @blackgoatpod, on Facebook at facebook.com/blackgoatpod/, and on instagram at @blackgoatpod. You can email us at letters@theblackgoatpodcast.com. You can subscribe to us on iTunes or Stitcher.

Our theme music is Peak Beak by Doctor Turtle, available on freemusicarchive.org under a Creative Commons noncommercial attribution license. Our logo was created by Jude Weaver.

This is episode 73. It was recorded on January 6, 2020.

In our annual end-of-year episode, we talk about noteworthy reflections and events from the year that just passed. Alexa reflects on breakups, and wonders why we don't take them more seriously as a significant disruption to other people's lives. Sanjay talks about hitting a low point and deciding to finally do something about it. And Simine talks about starting a new relationship and finding a new job that will take her halfway around the world. Plus: we answer a letter about whether scientists should mix advocacy and science.

The Black Goat is hosted by Sanjay Srivastava, Alexa Tullett, and Simine Vazire. Find us on the web at www.theblackgoatpodcast.com, on Twitter at @blackgoatpod, on Facebook at facebook.com/blackgoatpod/, and on instagram at @blackgoatpod. You can email us at letters@theblackgoatpodcast.com. You can subscribe to us on iTunes or Stitcher.

Our theme music is Peak Beak by Doctor Turtle, available on freemusicarchive.org under a Creative Commons noncommercial attribution license. Our logo was created by Jude Weaver.

This is episode 72. It was recorded on December 10, 2019.

Peer review is a major part of how science works today. In this episode we talk about how we approach doing peer reviews. How do you distinguish between differences in approach or preference - "I would have done it a different way" - versus things that you should treat as objections? How much weight do you put on different considerations - the importance of the research question, the novelty, the theory, the methods, the results, and other factors? What's your actual process - do you read front-to-back, or jump around? How much do you edit and wordsmith your reviews? When there are appendices, supplements, open code and materials, and preregistrations, which things do you read and how do you factor them in? How do you think about your potential biases and how to mitigate them? Plus: We answer a letter about deciding whether to pursue a postdoc versus other options.

The Black Goat is hosted by Sanjay Srivastava, Alexa Tullett, and Simine Vazire. Find us on the web at www.theblackgoatpodcast.com, on Twitter at @blackgoatpod, on Facebook at facebook.com/blackgoatpod/, and on instagram at @blackgoatpod. You can email us at letters@theblackgoatpodcast.com. You can subscribe to us on iTunes or Stitcher.

Our theme music is Peak Beak by Doctor Turtle, available on freemusicarchive.org under a Creative Commons noncommercial attribution license. Our logo was created by Jude Weaver.

This is episode 71. It was recorded on December 6, 2019.

To get your PhD you have to do a dissertation. For some this is an important product that demonstrates your ability to produce original research. To others, it's a vestigial ritual and a waste of time on the way to becoming a productive scholar. In this episode we discuss dissertations - what they've been in the past, what they are today, and where they might go in the future. Is a dissertation necessary for the kinds of work that someone might do with a PhD? As graduate training has evolved, how well has the dissertation kept up? Are oral defenses a valuable part of the process or an elaborate hazing ritual? Are they better if they're public, private, or don't happen at all? And most importantly, should all defenses involve swords? Plus: We discuss a letter about escaping from a toxic and abusive advisor.

The Black Goat is hosted by Sanjay Srivastava, Alexa Tullett, and Simine Vazire. Find us on the web at www.theblackgoatpodcast.com, on Twitter at @blackgoatpod, on Facebook at facebook.com/blackgoatpod/, and on instagram at @blackgoatpod. You can email us at letters@theblackgoatpodcast.com. You can subscribe to us on iTunes or Stitcher.

Our theme music is Peak Beak by Doctor Turtle, available on freemusicarchive.org under a Creative Commons noncommercial attribution license. Our logo was created by Jude Weaver.

This is episode 70. It was recorded on November 20, 2019.

Speaking up about injustice and bad behavior in a professional setting - as a witness, or as the target of it - is hard. It's uncomfortable, it's difficult, and it can generate backlash and other risks for yourself and your career. In this episode, we talk about that moment when people finally decide to say something or do something. Simine shares the story of how she decided to go on the record about being groped at a conference - what brought her to that decision, and what happened as a result. And we talk about other cases of people speaking up about harassment, discrimination, professional misconduct, and more, including Jennifer Freyd's pay discrimination lawsuit against the University of Oregon. We talk about the burden of knowing something is wrong, how this dilemma often falls disproportionately on people who are vulnerable in other ways, and what factors can help somebody speak out. Plus: we respond to a letter about department leaders who are obsessed with bean-counting of grant dollars and impact factors.

Links:

• N-best evaluation for academic hiring and promotion, by Michael Frank
• Making research evaluation more transparent: Aligning research philosophy, institutional values, and reporting, by Michael Dougherty, L. Robert Slevc, and James Grand, published at Perspectives in Psychological Science
• Dan Engber's Slate article where Simine went on the record
• What Reporting Sexual Harassment Taught Me, by Simine Vazire, published at Slate
• a little bit louder now, by Simine Vazire
• Taylor Swift’s Sexual Assault Testimony Was Sharp, Gutsy, and Satisfying
• Coverage of Jennifer Freyd's lawsuit: Psychology professor appeals dismissal in equal pay lawsuit with UO, Daily Emerald; and 47 Women's And Civil Rights Groups Support Equal Pay Lawsuit Against UO, OPB
• Why We Find and Expose Bad Science, by James Heathers

The Black Goat is hosted by Sanjay Srivastava, Alexa Tullett, and Simine Vazire. Find us on the web at www.theblackgoatpodcast.com, on Twitter at @blackgoatpod, on Facebook at facebook.com/blackgoatpod/, and on instagram at @blackgoatpod. You can email us at letters@theblackgoatpodcast.com. You can subscribe to us on iTunes or Stitcher.

Our theme music is Peak Beak by Doctor Turtle, available on freemusicarchive.org under a Creative Commons noncommercial attribution license. Our logo was created by Jude Weaver.

This is episode 69. It was recorded on October 30, 2019.