Ioannidis, J. P. A. (2005). Why Most Published Research Findings Are False. PLoS Medicine, 2(8), e124. https://doi.org/10.1371/journal.pmed.0020124

Neher, A. (1967). Probability Pyramiding, Research Error and the Need for Independent Replication. The Psychological Record, 17(2), 257–262. https://doi.org/10.1007/BF03393713

Moonesinghe, R., Khoury, M. J., & Janssens, A. C. J. W. (2007). Most Published Research Findings Are False—But a Little Replication Goes a Long Way. PLOS Medicine, 4(2), e28. https://doi.org/10.1371/journal.pmed.0040028

Stroebe, W. (2016). Are most published social psychological findings false? Journal of Experimental Social Psychology, 66, 134–144. https://doi.org/10.1016/j.jesp.2015.09.017

Diekmann, A. (2011). Are Most Published Research Findings False? Jahrbücher Für Nationalökonomie Und Statistik, 231(5–6), 628–635. https://doi.org/10.1515/jbnst-2011-5-606

Goodman, S., & Greenland, S. (2007). Why most published research findings are false: Problems in the analysis. PLoS Medicine, 4(4), e168.

Ioannidis, J. P. A. (2007). Why most published research findings are false: Author’s reply to Goodman and Greenland. PLoS Medicine, 4(6), e215.

In preparation for our discussion of "Why Most Published Research Findings Are False" by John Ioannidis from 2005, we read a very similar paper published 40 years earlier:

Neher, A. (1967). Probability Pyramiding, Research Error and the Need for Independent Replication. The Psychological Record, 17(2), 257–262. https://doi.org/10.1007/BF03393713

How I Fail. Blog by Veronika Cheplygina https://veronikach.com/category/how-i-fail/

Arkin, R. (2011). Most Underappreciated: 50 Prominent Social Psychologists Describe Their Most Unloved Work. Oxford University Press.

Kerr, N. L. (1998). HARKing: Hypothesizing After the Results are Known. Personality and Social Psychology Review, 2(3), 196–217. https://doi.org/10.1207/s15327957pspr0203_4

Sharpe, D. (2013). Why the resistance to statistical innovations? Bridging the communication gap. Psychological Methods, 18(4), 572–582. https://doi.org/10.1037/a0034177

Anti-Creativity Letters episode: https://nulliusinverba.podbean.com/e/prologus-23-the-anticreativity-letters-r-e-nisbett

Rouder, J. N., Haaf, J. M., & Snyder, H. K. (2019). Minimizing Mistakes in Psychological Science. Advances in Methods and Practices in Psychological Science, 2(1), 3–11. https://doi.org/10.1177/2515245918801915

Firestein, S. (2015). Failure: Why Science Is So Successful (First Edition). Oxford University Press.

In this episode, we delve into induction and deduction and talk further about issues related to generalizability. 

Shownotes

• Popper, K. The Logic of Scientific Discovery. (1953). Hutchinson &  Co. (Originally published in 1935)
• Yarkoni, T. (2022). The generalizability crisis. Behavioral and Brain Sciences, 45, e1.
• Mook, D. G. (1983). In defense of external invalidity. American psychologist, 38(4), 379-387.
• Salmon, W. C. (1981). Rational Prediction. The British Journal for the Philosophy of Science, 32(2), 115–125. https://doi.org/10.1093/bjps/32.2.115
• Reichenbach, H. (1938) [2006], Experience and Prediction: An Analysis of the Foundations and the Structure of Knowledge, Chicago: University of Chicago Press.
• Senn, S. (2007). Statistical issues in drug development (2nd ed). John Wiley & Sons.
• Ernst, M. D. (2004). Permutation Methods: A Basis for Exact Inference. Statistical Science, 19(4), 676–685. 
• Bacon, F. (1620). Instauratio magna [Novum organum]. London: John Bill.
• Urbach, P. (1982). Francis Bacon as a Precursor to Popper. The British Journal for the Philosophy of Science, 33(2), 113–132.

In this episode, we discuss the paper "In defense of external invalidity" by Douglas Mook. 

Shownotes

• Mook, D. G. (1983). In defense of external invalidity. American Psychologist, 38(4), 379–387.
• Mook, D. G. (1989). The myth of external validity. Everyday cognition in adulthood and late life, 25-43.
• The case of Phineas Gage was written up: Harlow, J. M. (1848). Passage of an iron rod through the head. The Boston Medical and Surgical Journal (1828-1851), 39(20)

A reading of the paper In Defense of External Invalidty by Douglas G. Mook, which will be discussed in the next episode. 

• Mook, D. G. (1983). In defense of external invalidity. American Psychologist, 38(4), 379–387. https://doi.org/10.1037/0003-066X.38.4.379

In this episode, we discuss the role of apprenticeship in training scientists and researchers. What’s the difference between traditional apprenticeship and cognitive apprenticeship? Does graduate training live up to its promise as an apprenticeship model? What can we do to improve the modeling of skills that are to be taught during graduate training? 

Shownotes

• Collins, A., Brown, J. S., & Holum, A. (1991). Cognitive apprenticeship: Making thinking visible. American educator, 15(3), 6-11.
• Gabrys, B. J., & Beltechi, A. (2012). Cognitive apprenticeship: The making of a scientist. In Reshaping doctoral education (pp. 144-155). Routledge.
• Casadevall, A., & Fang, F. C. (2016). Rigorous science: a how-to guide. MBio, 7(6), 10-1128.
• Alvesson, M., Gabriel, Y., & Paulsen, R. (2017). Return to meaning: A social science with something to say. Oxford University Press.
• Polanyi, M. (1958). Personal Knowledge: Towards a Post-Critical Philosophy (M. J. Nye, Ed.). University of Chicago Press.

This is a live episode, recorded in Växjö, Sweden (Linnaeus university) on September 24, 2024, at the 5th meeting of the Open Science Community Sweden and the Swedish Reproducibility Network. Thanks to André Kalmendal at Mono (https://monovaxjo.se) for recording the episode. 

In this episode, we discuss the paper "A case history in scientific method" by B. F. Skinner

Shownotes

• Skinner, B. F. (1956). A case history in scientific method. American psychologist, 11(5), 221.
• Richter, C. P. (1953). Free research versus design research. Science, 118(3056), 91–93.
• https://archive.org/details/WaldenTwoChapter01

In preparation for the next episode, in which we discuss this paper, here is a reading of: 

In today’s episode, we discuss critically reading and appraising scientific articles. How do we select which articles to read carefully? Which heuristics are useful for assessing paper quality? And do open science practices actually lead to better quality papers? Enjoy. 

Shownotes

• Bacon, F. (1625). Of Studies. 
• PNAS Submissions contributed by NAS members "The contributing member submits the manuscript to PNAS along with the names of at least two experts in the field of the paper who have agreed to review the work and brief comments about why each of those reviewers was chosen."  https://www.pnas.org/pb-assets/authors/ifora-1720190309383.pdf 
• How many p-values just below 0.05 should we expect across multiple tests?  https://daniellakens.blogspot.com/2014/05/the-probability-of-p-values-as-function.html 
• Lakens, D. (2024). When and How to Deviate From a Preregistration. Collabra: Psychology, 10(1), 117094. https://doi.org/10.1525/collabra.117094 
• TIER protocol: https://www.projecttier.org/tier-protocol/protocol-4-0/ 
• Gino fraud investigation and excel meta-data: https://datacolada.org/109 
• REAPPRAISED checklist: https://www.nature.com/articles/d41586-019-03959-6 
• Yang, Z., & Hung, I. W. (2021). Creative thinking facilitates perspective taking. Journal of Personality and Social Psychology, 120(2), 278.

In this episode, we talk about academic societies, professional organizations, and academic advocacy groups, focusing primarily on the discipline of psychology. What are their roles and responsibilities? Is it necessary for researchers to join such organizations? And should we bring back scholarly soirees? Enjoy. 

Shownotes

• Royal Society Referee Reports

• Psychological Science

• APA Divisions

• Consistori del Gay Saber

• ReproducibiliTea

• The Royal Society Soirées

In this episode, we discuss what separates science from pseudoscience and touch upon the demarcation problem, the recent controversial podcast called the Telepathy Tapes, and the movie Ghostbusters. Enjoy. 

Shownotes

• McLean v. Arkansas 
• Pigliucci, M., & Boudry, M. (Eds.). (2019). Philosophy of pseudoscience: Reconsidering the demarcation problem. University of Chicago Press.
• Report of the Royal Commission to Investigate Animal Magnetism
• The Telepathy Tapes
• Frankfurt, H. G. (2009). On bullshit.
• Moberger, V. (2020). Bullshit, pseudoscience and pseudophilosophy. Theoria, 86(5), 595–611.
• Ghostbusters (1984) - Venkman's ESP Test Scene
•  

In this episode, we discuss review boards for research with human subjects. Are they necessary? Are they efficient? Are scientists well equipped to make judgements about ethics? And are economists more ethical than psychologists? 

Shownotes

• Whitney, S. N. (2015). Balanced ethics review: A guide for institutional review board members. Springer.
• Schrag, Z. M. (2010). Ethical imperialism: Institutional review boards and the social sciences, 1965–2009. JHU Press.
• Kinsey Reports
  • Masters & Johnson
• How Institutional Review Boards can be (and are) Weaponized Against Academic Freedom
• Weaponizing the IRB 2.0
• Psychologists’ Involvement in Torture and the APA. Psychology Today.

In this episode, we discuss activism in science. How do political and personal values affect science? When is activism just part of the job? And should one be careful about activism in the classroom? Enjoy.  

Shownotes: 

• Frisby, C. L., Redding, R. E., & O’Donohue, W. T. (2023). Ideological and Political Bias in Psychology: An Introduction. In Ideological and Political Bias in Psychology: Nature, Scope, and Solutions (pp. 1-14). Cham: Springer International Publishing.
• McCaughey, M. (2023). The Trouble with Scholar-Activists. AAUP. 
• McCaughey, M. (2024). Against Scholar Activists. The James G. Martin Center for Academic Renewal. 
• Honeycutt, N., & Jussim, L. (2023). Political bias in the social sciences: A critical, theoretical, and empirical review. Ideological and Political Bias in Psychology: Nature, Scope, and Solutions, 97-146.
• Sargent, R. M. (2012). From Bacon to Banks: The vision and the realities of pursuing science for the common good. Studies in History and Philosophy of Science Part A, 43(1), 82-90.
• Weber, M. (1946). Science as a Vocation. In Science and the Quest for Reality (pp. 382-394). London: Palgrave Macmillan UK.

In this episode, we continue our discussion of replications. We talk about how to analyze replication studies, which studies are worth replicating, and what is the status of replications in other scientific disciplines. 

Shownotes

• Mack, R. W. (1951). The Need for Replication Research in Sociology. American Sociological Review, 16(1), 93–94. https://doi.org/10.2307/2087978
• Smith, N. C. (1970). Replication studies: A neglected aspect of psychological research. American Psychologist, 25(10), 970–975. https://doi.org/10.1037/h0029774
• Sidman, M. (1960). Tactics of Scientific Research: Evaluating Experimental Data in Psychology (New edition). Cambridge Center for Behavioral.
• Ebersole, C. R., Mathur, M. B., Baranski, E., Bart-Plange, D.-J., Buttrick, N. R., Chartier, C. R., Corker, K. S., Corley, M., Hartshorne, J. K., IJzerman, H., Lazarević, L. B., Rabagliati, H., Ropovik, I., Aczel, B., Aeschbach, L. F., Andrighetto, L., Arnal, J. D., Arrow, H., Babincak, P., … Nosek, B. A. (2020). Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability. Advances in Methods and Practices in Psychological Science. https://doi.org/10.1177/2515245920958687
• Isager, P. M., van Aert, R. C. M., Bahník, Š., Brandt, M. J., DeSoto, K. A., Giner-Sorolla, R., Krueger, J. I., Perugini, M., Ropovik, I., van ’t Veer, A. E., Vranka, M., & Lakens, D. (2023). Deciding what to replicate: A decision model for replication study selection under resource and knowledge constraints. Psychological Methods, 28(2), 438–451. https://doi.org/10.1037/met0000438
• Aldhous, P. (2011). Journal rejects studies contradicting precognition. New Scientist. https://www.newscientist.com/article/dn20447-journal-rejects-studies-contradicting-precognition/
• Stanley, D. J., & Spence, J. R. (2014). Expectations for Replications: Are Yours Realistic? Perspectives on Psychological Science, 9(3), 305–318. https://doi.org/10.1177/1745691614528518
• Simonsohn, U. (2015). Small telescopes: Detectability and the evaluation of replication results. Psychological Science, 26(5), 559–569. https://doi.org/10.1177/0956797614567341
• Nosek, B.A., Errington, T.M. (2017) Reproducibility in Cancer Biology: Making sense of replications. eLife 6:e23383. https://doi.org/10.7554/eLife.23383

In the next two episodes, we will discuss replication studies, which are essential to building reliable scientific knowledge.

Shownotes

• Mack, R. W. (1951). The Need for Replication Research in Sociology. American Sociological Review, 16(1), 93–94. https://doi.org/10.2307/2087978
• Smith, N. C. (1970). Replication studies: A neglected aspect of psychological research. American Psychologist, 25(10), 970–975. https://doi.org/10.1037/h0029774
• Sidman, M. (1960). Tactics of Scientific Research: Evaluating Experimental Data in Psychology (New edition). Cambridge Center for Behavioral.
• Ebersole, C. R., Mathur, M. B., Baranski, E., Bart-Plange, D.-J., Buttrick, N. R., Chartier, C. R., Corker, K. S., Corley, M., Hartshorne, J. K., IJzerman, H., Lazarević, L. B., Rabagliati, H., Ropovik, I., Aczel, B., Aeschbach, L. F., Andrighetto, L., Arnal, J. D., Arrow, H., Babincak, P., … Nosek, B. A. (2020). Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability. Advances in Methods and Practices in Psychological Science. https://doi.org/10.1177/2515245920958687
• Isager, P. M., van Aert, R. C. M., Bahník, Š., Brandt, M. J., DeSoto, K. A., Giner-Sorolla, R., Krueger, J. I., Perugini, M., Ropovik, I., van ’t Veer, A. E., Vranka, M., & Lakens, D. (2023). Deciding what to replicate: A decision model for replication study selection under resource and knowledge constraints. Psychological Methods, 28(2), 438–451. https://doi.org/10.1037/met0000438
• Aldhous, P. (2011). Journal rejects studies contradicting precognition. New Scientist. https://www.newscientist.com/article/dn20447-journal-rejects-studies-contradicting-precognition/
• Stanley, D. J., & Spence, J. R. (2014). Expectations for Replications: Are Yours Realistic? Perspectives on Psychological Science, 9(3), 305–318. https://doi.org/10.1177/1745691614528518
• Simonsohn, U. (2015). Small telescopes: Detectability and the evaluation of replication results. Psychological Science, 26(5), 559–569. https://doi.org/10.1177/0956797614567341

In this episode, we discuss a fun mix of eponymous laws, which are laws named after individuals who postulate them. 

Shownotes

• Campbell, D. T. (1979). Assessing the impact of planned social change. Evaluation and Program Planning, 2(1), 67–90. https://doi.org/10.1016/0149-7189(79)90048-X
• Merton, R. K. (1995). The Thomas Theorem and the Matthews Effect. Social Forces, 74(2), 379–422.
• Stigler, S. M. (1980). Stigler’s Law of Eponymy*. Transactions of the New York Academy of Sciences, 39(1 Series II), 147–157. https://doi.org/10.1111/j.2164-0947.1980.tb02775.x
• Clarke, A. C. (Arthur C. (1962). Profiles of the future: An inquiry into the limits of the possible. New York : Bantam Books. http://archive.org/details/profilesoffuture00clar
• Brandolini’s Law: Based on a tweet, after reading Kahneman Thinking fast and slow: https://twitter.com/ziobrando/status/289635060758507521
• Preston, I. L. (1980). Researchers at the Federal Trade Commission—Peril and Promise. Current Issues and Research in Advertising, 3(1), 1–15.
• Twyman’s Law: “The more unusual or interesting the data, the more likely they are to have been the result of an error of one kind or another.” Earliest scholarly reference is in Journal of the Royal Statistical Society, Series A, Vol 138, No 4, 1975. The Teaching of Statistics by A. S. C. Ehrenberg.
• Bloch, A. (1990). Murphy's law complete: All the reasons why everything goes wrong. Arrow Books Limited. 

In this final episode of the three-part series on the Philosophical Psychology lectures by Paul Meehl, we discuss lectures 6-8, which cover the ten obfuscating factors in "soft areas" of psychology and a host of advice Meehl provides for researchers, reviewers, editors, and educators on how to improve practice. 

Shownotes

• Krefeld-Schwalb, A., Sugerman, E. R., & Johnson, E. J. (2024). Exposing omitted moderators: Explaining why effect sizes differ in the social sciences. Proceedings of the National Academy of Sciences, 121(12), e2306281121. https://doi.org/10.1073/pnas.2306281121
• Lakens, D., & Etz, A. J. (2017). Too True to be Bad: When Sets of Studies With Significant and Nonsignificant Findings Are Probably True. Social Psychological and Personality Science, 8(8), 875–881. https://doi.org/10.1177/1948550617693058

In this episode, we continue the discussion of Meehl's Philosophy of Psychology course, focusing on lectures 3, 4, and 5. 

Shownotes

• The quote "Don't make a mockery of honest ad-hockery" is probably from Clark Glymour: https://en.wikipedia.org/wiki/Clark_Glymour
• Good, I. J. (1965). The Estimation of Probabilities: An Essay on Modern Bayesian Methods. M.I.T. Press.
• Shepard, R. N. (1987). Toward a universal law of generalization for psychological science. Science, 237(4820), 1317–1323.

Video lectures: https://meehl.umn.edu/video 

Faust, D., & Meehl, P. E. (1992). Using scientific methods to resolve questions in the history and philosophy of science: Some illustrations. Behavior Therapy, 23(2), 195–211. https://doi.org/10.1016/S0005-7894(05)80381-8

Serlin, R. C., & Lapsley, D. K. (1985). Rationality in psychological research: The good-enough principle. American Psychologist, 40(1), 73–83. https://doi.org/10.1037/0003-066X.40.1.73

Meehl, P. E. (1990). Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant it. Psychological Inquiry, 1(2), 108–141. https://doi.org/10.1207/s15327965pli0102_1

Meehl, P. E. (1992). Cliometric metatheory: The actuarial approach to empirical, history-based philosophy of science. Psychological Reports, 71, 339–467.

Conspiracy Stories Show Notes: 

• Zeitgeist documentary: https://en.wikipedia.org/wiki/Zeitgeist_(film_series)
• Podcast Drang naar Samenhang: https://podcasts.apple.com/nl/podcast/drang-naar-samenhang/id1584797552
• This is not a conspiracy theory documentary. https://www.everythingisaremix.info/tinact 
• Parker, M. (2000). Human Science as Conspiracy Theory. The Sociological Review, 48(2_suppl), 191-207. https://doi.org/10.1111/j.1467-954X.2000.tb03527.x
• Douglas, K. M., Sutton, R. M., & Cichocka, A. (2017). The Psychology of Conspiracy Theories. Current Directions in Psychological Science, 26(6), 538-542. https://doi.org/10.1177/0963721417718261
• Tage-gate episode of More of a Comment than a Question:  https://moreofacomment.buzzsprout.com/1207223/episodes/5511751-tage-gate 

In advance of the next three episodes discussing the Philosophical Psychology lectures by Paul E. Meehl, we present a brief reading from his autobiography in A history of psychology in autobiography.

• Meehl, P. E. (1989). Paul E. Meehl. In G. Lindzey (Ed.), A history of psychology in autobiography (Vol. 8, pp. 337–389). Stanford, CA: Stanford University Press.

In this episode, we discuss objectivity and disinterestedness in science. We talk about norms, values, interests, and objectivity in research practice, peer review, and hiring decisions. Is it possible to be completely objective? Is objectivity a feature of epistemic products or epistemic processes? And most importantly, how would you objectively rate this podcast?

Shownotes

• Armstrong, J. S. (1979). Advocacy and objectivity in science. Management Science, 25(5), 423–428.
• Declaration of Interest by Stephen Senn: http://senns.uk/Declaration_Interest.htm
• Djørup, S., & Kappel, K. (2013). The norm of disinterestedness in science; a restorative analysis. SATS, 14(2). https://doi.org/10.1515/sats-2013-0009
• Elliott, K. C. (2017). A Tapestry of Values: An Introduction to Values in Science. Oxford University Press. https://doi.org/10.1093/acprof:oso/9780190260804.001.0001
• Feyerabend, Paul. "How to defend society against science." Philosophy: Basic Readings (1975): 261-271.
• Jamieson, K. H., McNutt, M., Kiermer, V., & Sever, R. (2019). Signaling the trustworthiness of science. Proceedings of the National Academy of Sciences, 116(39), 19231–19236. https://doi.org/10.1073/pnas.1913039116
• Janack, M. (2002). Dilemmas of objectivity. Social Epistemology, 16(3), 267-281.
• John, S. (2021). Objectivity in science. Cambridge University Press.
• Merton, R. K. (1973). The sociology of science: Theoretical and empirical investigations. University of Chicago Press.
• Mitroff, I. I. (1974). Norms and Counter-Norms in a Select Group of the Apollo Moon Scientists: A Case Study of the Ambivalence of Scientists. American Sociological Review, 39(4), 579–595. https://doi.org/10.2307/2094423
• Mitroff, I. I. (1974). The subjective side of science: A philosophical inquiry into the psychology of the Apollo moon scientists (First Edition). Elsevier.
• A Russian polar researcher has been charged trying to stab a colleague to death at a remote Antarctic base https://www.businessinsider.com/sergey-savitsky-alleged-attempted-murder-at-antarctic-bellingshausen-2018-10 
• Stamenkovic, P. (2023). Facts and objectivity in science. Interdisciplinary Science Reviews, 1–22. https://doi.org/10.1080/03080188.2022.2150807

In this episode, we discuss the role of criticism in science. When is criticism constructive as opposed to obsessive? What are the features of fair and useful scientific criticism? And should we explicitly teach junior researchers to both give and accept criticism?

Shownotes:

• Babbage, C. (1830). Reflections on the Decline of Science in England: And on Some of Its Causes.
• Prasad, Vinay, and John PA Ioannidis. "Constructive and obsessive criticism in science." European journal of clinical investigation 52.11 (2022): e13839.
• Lakatos, I. (1968, January). Criticism and the methodology of scientific research programmes. In Proceedings of the Aristotelian society (Vol. 69, pp. 149-186). Aristotelian Society, Wiley.
• LOWI: https://lowi.nl/en/home/ As an independent advisory body it plays a role in the complaints procedure about alleged violations of principles of research integrity.
• Holcombe, A. O. (2022). Ad hominem rhetoric in scientific psychology. British Journal of Psychology, 113(2), 434–454. https://doi.org/10.1111/bjop.12541
• Daniel C. Dennett: I've Been Thinking https://wwnorton.com/books/9780393868050 
• Phillip Stark textbook chapter on logical fallacies: https://www.stat.berkeley.edu/~stark/SticiGui/Text/reasoning.htm 
• Gelman, A., & Tuerlinckx, F. (2000). Type S error rates for classical and Bayesian single and multiple comparison procedures. Computational Statistics, 15(3), 373–390. https://doi.org/10.1007/s001800000040
• Popper, K. R. (1959). The logic of scientific discovery. Routledge.
• PubPeer: https://pubpeer.com

In this episode, we continue discussing Dubin’s 8-step method for theory building. We discuss the measurement of theoretical constructs, using logical propositions to make falsifiable predictions from theories, and the importance of specifying boundary conditions. 

Shownotes

• Jaccard, J., & Jacoby, J. (2010). Theory Construction and Model-building Skills: A Practical Guide for Social Scientists. Guilford Press.
• McGuire, W. J. (1973). The yin and yang of progress in social psychology: Seven koan. Journal of Personality and Social Psychology, 26(3), 446–456. https://doi.org/10.1037/h0034345
• Simons, D. J., Shoda, Y., & Lindsay, D. S. (2017). Constraints on Generality (COG): A Proposed Addition to All Empirical Papers. Perspectives on Psychological Science, 12(6), 1123–1128. https://doi.org/10.1177/1745691617708630
• Norm Macdonald: The Professor of Logic
• Raven Paradox: https://platonicrealms.com/encyclopedia/Hempels-Ravens-Paradox 
• Pavlov, I. (1936). Bequest of Pavlov to the Academic Youth of His Country. Science, 83(2155), 369–370. https://doi.org/10.1126/science.83.2155.369

In this episode we discussed the 8-step method of theory building proposed by Robin Dubin in his classic 1969 book Theory Building.

Shownotes

• Dubin, R. (1969). Theory building. Free Press. http://catalog.hathitrust.org/api/volumes/oclc/160506.html
• Lynham, S. A. (2002). Quantitative Research and Theory Building: Dubin’s Method. Advances in Developing Human Resources, 4(3), 242–276. https://doi.org/10.1177/15222302004003003
• Elms, A. C. (1975). The crisis of confidence in social psychology. American Psychologist, 30(10), 967.
• Meehl, P. E. (1978). Theoretical Risks and Tabular Asterisks: Sir Karl, Sir Ronald, and the Slow Progress of Soft Psychology. Journal of Consulting and Clinical Psychology, 46(4), 806–834. https://doi.org/10.1037/0022-006X.46.4.806
• Swedberg, R. (2014). The art of social theory. Princeton University Press.
• Ben Wright: https://en.wikipedia.org/wiki/Benjamin_Drake_Wright
• Yarkoni, T., & Westfall, J. (2017). Choosing Prediction Over Explanation in Psychology: Lessons From Machine Learning. Perspectives on Psychological Science, 12(6), 1100–1122. https://doi.org/10.1177/1745691617693393
• Isaac, M. G., Koch, S., & Nefdt, R. (2022). Conceptual engineering: A road map to practice. Philosophy Compass, 17(10), e12879. https://doi.org/10.1111/phc3.12879

In this episode, we discuss the barriers to cumulative science, including inconsistent measurement tools, overreliance on single studies, and the large volume of research publications. Can replications, interdisciplinary collaborations, and prospective meta-analyses help us solve this issue? Can AI solve all our problems?  And do most scientists treat their theories like toothbrushes?

Shownotes

• Opening quote by George Sarton
  • Sarton, G. (1927). Introduction to the History of Science (Vol. 376). 
• Is Science Cumulative? a Physicist Viewpoint: https://link.springer.com/chapter/10.1007/978-1-4020-6279-7_10
• Psychological Methods. (2009). Special Issue: Multi-Study Methods for Building a Cumulative Psychological Science.

• Walter Mischel, Becoming a Cumulative Science 

• Dorothy Bishop - Why we need cumulative science (AIMOS)
• Watkins, J. W. (1984). Science and Skepticism. Princeton University Press.

In today’s episode, we continue our conversation about preregistration. How flexible can we be when we preregister, without increasing flexibility in our analysis? How well do people preregister, and what does a good preregistration look like? And how do we deal with deviations from preregistrations?

Shownotes

• Dubin, R. (1969). Theory building. Free Press. His full quote is: "There is no more devastating commendation that the self-designated theorist makes of the researcher than to label his work purely descriptive". 
• Claesen, A., Gomes, S., Tuerlinckx, F., & Vanpaemel, W. (2021). Comparing dream to reality: An assessment of adherence of the first generation of preregistered studies. Royal Society Open Science, 8(10), 211037. https://doi.org/10.1098/rsos.211037
• Akker, O. van den, Bakker, M., Assen, M. A. L. M. van, Pennington, C. R., Verweij, L., Elsherif, M., Claesen, A., Gaillard, S. D. M., Yeung, S. K., Frankenberger, J.-L., Krautter, K., Cockcroft, J. P., Kreuer, K. S., Evans, T. R., Heppel, F., Schoch, S. F., Korbmacher, M., Yamada, Y., Albayrak-Aydemir, N., … Wicherts, J. (2023). The effectiveness of preregistration in psychology: Assessing preregistration strictness and preregistration-study consistency. MetaArXiv. https://doi.org/10.31222/osf.io/h8xjw
• Sequential analysis and alpha spending functions https://lakens.github.io/statistical_inferences/10-sequential.html 
• Bishop, D. V. M. (2018). Fallibility in Science: Responding to Errors in the Work of Oneself and Others. Advances in Methods and Practices in Psychological Science, 2515245918776632. https://doi.org/10.1177/2515245918776632
• FDAAA Trial Tracker https://fdaaa.trialstracker.net 
• Ensinck, E., & Lakens, D. (2023). An Inception Cohort Study Quantifying How Many Registered Studies are Published. PsyArXiv. https://doi.org/10.31234/osf.io/5hkjz
• Quantitude episode on preregistration https://quantitudepod.org/s3e07-in-defense-of-researcher-degrees-of-freedom/ 
• Lakens, D. (2023). When and How to Deviate from a Preregistration. PsyArXiv. https://doi.org/10.31234/osf.io/ha29k 

In this two part episode we discuss the fine art of preregistration. We go back into the history of preregistration, its evolution, and current use. Do we preregister to control the Type 1 error rate, or to show that we derived our prediction from theory a priori? Can and should we preregister exploratory or secondary data analysis? And how severe is the issue of severe testing?

Shownotes

• ClinicalTrials.gov
• You can preregister on AsPredicted and the OSF
• Johnson, M. (1975). Models of Control and Control of Bias. European Journal of Parapsychology, 36–44.
• SPIRIT Checklist
• Bishop, D. V. M. (2018). Fallibility in Science: Responding to Errors in the Work of Oneself and Others. Advances in Methods and Practices in Psychological Science, 1(3), 432–438. https://doi.org/10.1177/2515245918776632
• FDA trials tracker: https://fdaaa.trialstracker.net 
• Ensinck, E., & Lakens, D. (2023). An Inception Cohort Study Quantifying How Many Registered Studies are Published. PsyArXiv. https://doi.org/10.31234/osf.io/5hkjz
• van den Akker, O. R., van Assen, M. A. L. M., Enting, M., de Jonge, M., Ong, H. H., Rüffer, F., Schoenmakers, M., Stoevenbelt, A. H., Wicherts, J. M., & Bakker, M. (2023). Selective Hypothesis Reporting in Psychology: Comparing Preregistrations and Corresponding Publications. Advances in Methods and Practices in Psychological Science, 6(3), 25152459231187988. https://doi.org/10.1177/25152459231187988
• Claesen, A., Gomes, S., Tuerlinckx, F., & Vanpaemel, W. (2021). Comparing dream to reality: An assessment of adherence of the first generation of preregistered studies. Royal Society Open Science, 8(10), 211037. https://doi.org/10.1098/rsos.211037
• Bakan, D. (1966). The test of significance in psychological research. Psychological Bulletin, 66(6), 423–437. https://doi.org/10.1037/h0020412
• Rosenthal, R. (1966). Experimenter effects in behavioral research. Appleton-Century-Crofts.
• Johnson, M. (1975). Models of Control and Control of Bias. European Journal of Parapsychology, 36–44.
• de Groot, A. D. (1969). Methodology. Mouton & Co.
• Claesen, A., Lakens, D., Vanpaemel, W., & Dongen, N. van. (2022). Severity and Crises in Science: Are We Getting It Right When We’re Right and Wrong When We’re Wrong? PsyArXiv. https://doi.org/10.31234/osf.io/ekhc8

In the first episode of 2024, we discuss the double-edged sword: reverence to authority. Should scientists respect others on whose shoulders they stand? Or should they be wary of appeal to authority? How should scientists deal with other sources of authority in science, like for example, the government or academic societies? And how can we differentiate true expertise from mere authority?  Enjoy. 

Shownotes

• Frank, P. (1956). The role of authority in the interpretation of science. Synthese, 10, 335–338.
• Barber, B. (1952). Science and the social order. Glencoe, Ill. : Free Press. http://archive.org/details/sciencesocialord0000barb
• Barber, B. (1961). Resistance by Scientists to Scientific Discovery. Science, 134(3479), 596–602.
• Kitcher, P. (1992). Authority, deference, and the role of individual reasoning in science. In E. Mcmullin (Ed.), The social dimensions of science. Notre Dame: The University of Notre Dame Press
• Polanyi, M. (1962). The republic of science. Minerva, 1(1), 54–73
• The practice of two-spaces after the end of a sentence comes from when type-writers used monospaced typefaces: https://slate.com/technology/2011/01/two-spaces-after-a-period-why-you-should-never-ever-do-it.html 

Broad, W. J., & Wade, N. (1983). Betrayers of the truth. New York : Simon and Schuster. http://archive.org/details/betrayersoftruth00broa

Wolfgang Stroebe, Tom Postmes, & Russell Spears. (2012). Scientific Misconduct and the Myth of Self-Correction in Science. Perspectives on Psychological Science, 7(6), 670–688. https://doi.org/10.1177/1745691612460687

Zotero can track if you are citing retractions: https://retractionwatch.com/2019/06/12/want-to-check-for-retractions-in-your-personal-library-and-get-alerts-for-free-now-you-can/

100% CI blog: The Untold Mystery of Rogue RA https://www.the100.ci/2024/12/18/rogue-ra/

Merton, R. K. (1957). Priorities in Scientific Discovery: A Chapter in the Sociology of Science. American Sociological Review, 22(6), 635–659. https://doi.org/10.2307/2089193

Senior RIKEN scientist involved in stem cell scandal commits suicide https://www.science.org/content/article/senior-riken-scientist-involved-stem-cell-scandal-commits-suicide

Kis, A., Tur, E. M., Lakens, D., Vaesen, K., & Houkes, W. (2022). Leaving academia: PhD attrition and unhealthy research environments. PLOS ONE, 17(10), e0274976. https://doi.org/10.1371/journal.pone.0274976

The Fixation of Belief. Charles S. Peirce. Popular Science Monthly 12 (November 1877), 1-15.

• http://peirce.org/writings/p107.html 

In this second installment of The Anticreativity Letters, we continue discussing the Tempter's tactics for stifling creativity and how to overcome them. 

In the first of a two-part episode, we discuss The Anticreativity Letters by Richard Nisbett, in which a senior "tempter" advises a junior tempter on ways to prevent a young psychologist from being a productive and creative scientist.

• Nisbett, R. E. (1990). The anticreativity letters: Advice from a senior tempter to a junior tempter. American Psychologist, 45(9), 1078–1082.

• BMJ Christmas issue: https://www.bmj.com/about-bmj/resources-authors/article-types/christmas-issue
• Quote by Ira Glass: https://www.goodreads.com/quotes/309485-nobody-tells-this-to-people-who-are-beginners-i-wish

A reading of:

Nisbett, R. E. (1990). The anticreativity letters: Advice from a senior tempter to a junior tempter. American Psychologist, 45(9), 1078–1082. https://doi.org/10.1037/0003-066X.45.9.1078

In today’s episode, we discuss the role of mentorship in academia. What are the characteristics of a good mentor-mentee relationship? What are the qualities of good mentors and good mentees? Does mentorship play a role in the development of scientific knowledge? And could mentors and mentees benefit from couples therapy?

Note: D.I.H.C is pronounced 'dick' but this is meant to be a family-friendly podcast :)

Shownotes

• https://www.psychologytoday.com/us/blog/emotional-fitness/201303/10-things-your-relationship-needs-thrive

• Roberts, L. R., Tinari, C. M., & Bandlow, R. (2019). An effective doctoral student mentor wears many hats and asks many questions. International Journal of Doctoral Studies, 14, 133.

• Sarabipour, S., Niemi, N. M., Burgess, S. J., Smith, C. T., Filho, A. W. B., Ibrahim, A., & Clark, K. (2023). Insights from a survey of mentorship experiences by graduate and postdoctoral researchers (p. 2023.05.05.539640). bioRxiv. https://doi.org/10.1101/2023.05.05.539640

In this episode, we discuss the role of trust in science. Why should we verify but trust other scientists? What are the prerequisites for building trust within the scientific community? Who is ultimately responsible for verifying our claims and practices that bolster those claims? And should we give personality tests to everyone who enters academia?

Shownotes

• Hardwig, J. (1991). The role of trust in knowledge. The Journal of Philosophy, 88(12), 693–708.

• Hendriks, F., Kienhues, D., Bromme, R. (2016). Trust in Science and the Science of Trust. In: Blöbaum, B. (eds) Trust and Communication in a Digitized World. Progress in IS. Springer, Cham. https://doi.org/10.1007/978-3-319-28059-2_8 

• Strand, J. F. (2023). Error tight: Exercises for lab groups to prevent research mistakes. Psychological Methods, No Pagination Specified-No Pagination Specified. https://doi.org/10.1037/met0000547

• Duygu Uygun-Tunç: Trust and criticism in science, Part I: Critical rationalism instead of organized skepticism: https://uyguntunc.wordpress.com/2020/10/30/trust-and-criticism-in-science-part-i-critical-rationalism-instead-of-organized-skepticism/

• Vazire, S. (2017). Quality Uncertainty Erodes Trust in Science. Collabra: Psychology, 3(1), 1. https://doi.org/10.1525/collabra.74

• Wicherts, J. M. (2011). Psychology must learn a lesson from fraud case. Nature, 480(7375), Article 7375. https://doi.org/10.1038/480007a

• Fricker, E. (2002). Trusting others in the sciences: A priori or empirical warrant? Studies in History and Philosophy of Science Part A, 33(2), 373–383. https://doi.org/10.1016/S0039-3681(02)00006-7

In advance of our episode Verify but Trust, a reading of John Hardwig's paper The Role of Trust in Science. 

• Hardwig, J. (1991). The role of trust in knowledge. The Journal of Philosophy, 88(12), 693–708.

In today’s episode, we discuss the peer review process---its history, its present, and its future. How does peer review work? How long has it existed in its current form? Should reviews be open and signed? Should reviewers be paid for their hard labor? Should we just abandon the peer review process, or does it have a positive role to play? 

Shownotes

• Peer Community in Registered Reports: https://rr.peercommunityin.org/

• Suggestion to Darwin to publish a book about pigeons instead of The Origins of Species: https://www.darwinproject.ac.uk/letter/DCP-LETT-2457A.xml

• Baldwin, M. (2018). Scientific Autonomy, Public Accountability, and the Rise of “Peer Review” in the Cold War United States. Isis, 109(3), 538–558. https://doi.org/10.1086/700070

• Burnham, J. C. (1990). The evolution of editorial peer review. JAMA, 263(10), 1323–1329.

In this episode, we discuss Quantifauxcation, described by statistician Philip Stark as “situations in which a number is, in effect, made up, and then is given credence merely because it is quantitative.” We give examples of quantifauxcation in psychology, including errors of the third kind. We spend the second half of the podcast discussing how to develop quantitative measures that are meaningful and bridge the divide between qualitative and quantitative observations.

Shownotes

• Statistics textbook by Philip Stark.

• Stark, P. B. (2022). Pay No attention to the model behind the curtain. Pure and Applied Geophysics, 179(11), 4121–4145. https://doi.org/10.1007/s00024-022-03137-2 

• Burgess, E. W. (1927). Statistics and case studies as methods of sociological research, Vol 12(3), 103-120. (Thanks to Andy Grieve!)

• Nick Brown's role in pointing out flaws in the positivity ratio.

• Retraction notice of the positivity ratio paper.

• Blog by Tania Lombrozo on nonsensical formulas in abstracts.

• Kimball, A. W. (1957). Errors of the third kind in statistical consulting. Journal of the American Statistical Association, 52(278), 133–142. https://doi.org/10.1080/01621459.1957.10501374

• Type III errors: Philip Stark’s post of Deborah Mayo’s blog

• Brower, D. (1949). The problem of quantification in psychological science. Psychological Review, 56(6), 325–333. https://doi.org/10.1037/h0061802

• Guttman scales

• Wilson, M. (2023). Constructing measures: An item response modeling approach. Taylor & Francis.

• Wilson, M., Bathia, S., Morell, L., Gochyyev, P., Koo, B. W., & Smith, R. (2022). Seeking a better balance between efficiency and interpretability: Comparing the likert response format with the Guttman response format. Psychological Methods. Advance online publication. https://doi.org/10.1037/met0000462

• Bhatti, H.A., Mehta, S., McNeil, R., Wilson, M. (2023). A scientific approach to assessment: Rasch measurement and the four building blocks. In X. Liu & W. Boone (Eds.), Advances in Applications of Rash Measurement in Science Education. Springer Nature. 

In preparation for a discussion on Quantifauxcation, a reading of 'Problem-Centering vs. Means-Centering in Science' by Abraham H. Maslow (1946). 

• Maslow, A. H. (1946). Problem-Centering vs. Means-Centering in Science. Philosophy of Science, 13(4), 326–331. https://doi.org/10.1086/286907

Babbage, C. (1830). Reflections on the Decline of Science in England: And on Some of Its Causes. B. Fellowes.

Sokal, A. D. (1996). Transgressing the Boundaries: Toward a Transformative Hermeneutics of Quantum Gravity. Social Text, 46/47, 217. https://doi.org/10.2307/466856

Grievance studies: https://en.wikipedia.org/wiki/Grievance_studies_affair

It is legal to own and/or read Mein Kampf in The Netherlands (and Germany).

Hand, D. (2007). Deception and dishonesty with data: Fraud in science. Significance, 4(1), 22–25. https://doi.org/10.1111/j.1740-9713.2007.00215.x

Gross, C. (2016). Scientific Misconduct. Annual Review of Psychology, 67(Volume 67, 2016), 693–711. https://doi.org/10.1146/annurev-psych-122414-033437

Paolo Macchiarini: https://www.science.org/content/article/macchiarini-guilty-misconduct-whistleblowers-share-blame-new-karolinska-institute

The Truth about China’s Cash-for-Publication Policy: https://www.technologyreview.com/2017/07/12/150506/the-truth-about-chinas-cash-for-publication-policy/

Claudine Gay plagiarism: https://www.plagiarismtoday.com/2024/01/22/harvard-releases-details-of-claudine-gay-investigation/

Many Co-Authors: https://manycoauthors.org/

Paper describing a replication study where students make up data: Azrin, N. H., Holz, W., Ulrich, R., & Goldiamond, I. (1961). The control of the content of conversation through reinforcement. Journal of the Experimental Analysis of Behavior, 4, 25–30.

Francesca Gino defamation case dismissed: https://www.thecrimson.com/article/2024/9/12/judge-dismisses-gino-lawsuit-defamation-charges/

Retractions in Social Influence of the work of Guéguen: https://www.tandfonline.com/doi/full/10.1080/15534510.2024.2431408, https://www.tandfonline.com/doi/full/10.1080/15534510.2024.2431415, https://www.tandfonline.com/doi/full/10.1080/15534510.2024.2431421

Diederik Stapel’s book: http://nick.brown.free.fr/stapel/FakingScience-20161115.pdf

Merton, R. K. (1957). Priorities in Scientific Discovery: A Chapter in the Sociology of Science. American Sociological Review, 22(6), 635–659. https://doi.org/10.2307/2089193

In today’s episode, we discuss intellectual vices. How can we tell the difference between justified confidence and unjustified arrogance? How do we deal with feelings of envy or negative comparison with other scientists? What is the difference between building one’s career and careerism? And what do we do about scientists who do not care about the truth? 

Shownotes

• Azrin, N. H., Holz, W., Ulrich, R., & Goldiamond, I. (1961). The control of the content of conversation through reinforcement. Journal of the Experimental Analysis of Behavior, 4, 25–30. https://doi.org/10.1901/jeab.1961.4-25 

• Meehl, P. E. (1967). Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science, 103–115. https://doi.org/10.1086/288135  

• Mitroff, I. I. (1974). Norms and Counter-Norms in a Select Group of the Apollo Moon Scientists: A Case Study of the Ambivalence of Scientists. American Sociological Review, 39(4), 579–595. https://doi.org/10.2307/2094423

• Susan Blackmore: https://en.wikipedia.org/wiki/Susan_Blackmore 

In this episode, we discuss scientific snobbery and the ways in which it affects our interactions with and perceptions of other scientists. What are the reasons for hierarchies among different disciplines, institutions, and approaches to science? What are some ways in which snobbery manifests in science? And is it snobby to not want to present scientific posters? Enjoy. 

Shownotes: 

• Ego and Math (3Blue1Brown)

• M. V. Berry; Regular and irregular motion. AIP Conf. Proc. 15 September 1978; 46 (1): 16–120. https://doi.org/10.1063/1.31417

In this episode, we continue our conversation on the replication crisis⏤Which methodological, theoretical, and practical concerns did psychologists raise half a century ago? What has changed, and what remains the same, during the current crisis? 

Shownotes

• Orne, M. T. (1962). On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. American Psychologist, 17(11), 776–783. https://doi.org/10.1037/h0043424

• Rosenthal, R. (1966). Experimenter effects in behavioral research. Appleton-Century-Crofts.

• Gergen, K. J. (1973). Social psychology as history. Journal of Personality and Social Psychology, 26, 309–320. https://doi.org/10.1037/h0034436

• Koole, S. L., & Lakens, D. (2012). Rewarding replications: A sure and simple way to improve psychological science. Perspectives on Psychological Science, 7(6), 608–614. https://doi.org/10.1177/1745691612462586

• Greenwald, A. G. (Ed.). (1976). An editorial. Journal of Personality and Social Psychology, 33(1), 1–7. https://doi.org/10.1037/h0078635

• Ring, K. (1967). Experimental social psychology: Some sober questions about some frivolous values. Journal of Experimental Social Psychology.

• Ledgerwood, A., & Sherman, J. W. (2012). Short, sweet, and problematic? The rise of the short report in psychological science. Perspectives on Psychological Science, 7(1), 60–66.

• Barber, T. X. (1976). Pitfalls in Human Research: Ten Pivotal Points. Pergamon Press.

• Dunnette, M. D. (1966). Fads, fashions, and folderol in psychology. American Psychologist, 21(4), 343–352. https://doi.org/10.1037/h0023535

• Babbage, C. (1830). Reflections on the Decline of Science in England: And on Some of Its Causes. B. Fellowes.

Reading of the chapter "Investigator Data Analysis Effect" from the book: