Critical Psychology Teaching Materials

Methods, Design, & Analysis

R.A. Josephs

Department of Psychology
University of Texas-Austin

Spring 1997
Psychology 391P

(submitted with permission by Tor Neilands)


This course serves as an introduction to the variety of issues and meta-issues surrounding scientific inquiry, with a strong emphasis on the social and behavioral sciences, and psychology in particular. The essence of science lies in its methodology, and the fate of any particular scientific endeavor is, in large part, a function of the quality of the associated methodology. Methodology is as much a way of thinking as it is a set of specific rules, guidelines, and techniques. This is not a "how to" course. Rather, it is a course that I hope will provide a number of important conceptual building blocks that can be used to help guide in your future inquiries.

At least 12 hours prior to each week's meeting, each of you will be required to hand in at least two substantive questions based on that week's readings. Questions will serve as a catalyst for that week's discussion. Send the questions via electronic mail to JOSEPHS@mail.utexas.edu

* Note - Those selections designated with an "*" are from: Tweeney, R.D., Doherty, M.E., & Mynatt, C.R. (Eds.). On Scientific Thinking. New York: Columbia University Press, 1981.

Section 1- Hypothesis Generation

Kuhn, T.S. The Structure of Scientific Revolutions. Chaps 2-4.

* Introduction (Hypothesis Generation)

* Selection 7--Isaac Newton "The Rules of Hypothesizing."

* Selection 9--Karl Popper "The Myth of Inductive Hypothesis Generation."

Masterman, M. The nature of a paradigm. In I. Lakatos and A. Musgrave (Eds.), Criticism and the Growth of Knowledge. Cambridge: Cambridge University Press, 1970.

McGuire, W.J. The Ying and Yang of progress in social psychology: Seven koan. Journal of Personality and Social Psychology, 1973, 26, 446-456.

Section 2 - Hypothesis Testing I

*Introduction (Hypothesis Testing)

* Selection 12--Karl Popper "Science, Pseudo-Science, and Falsifiability."

Crease, R.P. The trajectory of techniques: Lessons from the past. Science, 1992, 257, 350-353.

Hall, S. How technique is changing science. Science, 1992, 257, 344-349.

Chamberlin, T. C. The method of multiple working hypotheses. Journal of Geology, 1897, 5, 837-48.

Platt, J. R. Strong Inference. Science, 1964,146(3642), pp. 347-353.

* Selection 14--John C. Eccles "In Praise of Falsification."

* Selection 15--Robert Sekuler "In Praise of Strong Inference."

Section 3 - Hypothesis Testing II

*Introduction (Hypothesis Testing)

Mahoney, M.J. Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research, 1977, 1, 27-31.

Sterling, T.D., Rosenbaum, W.L., & Weinkam, J.J. Publication decisions revisited: The effect of outcome of statistical tests on the decision to publish and vice versa. The American Statistician, 1995, 49, 108-112.

Greenwald, A. G., Pratkanis, A. R., Leippe, M. R., & Baumgardner, M. H. Under what conditions does theory obstruct research progress? Psychological Review, 1986,93(2) 216-229.

Greenberg, J., Solomon, S., Pyszczynski, T., & Steinberg, L. A reaction to Greenwald, Pratkanis, Leippe, and Baumgardner (1986): Under what conditions does research obstruct theory progress? Psychological Review, 1988,95(4) 566-571

*Selection 20--Ian I. Mitroff "Scientists and Confirmation Bias."

*Selection 21--Howard E. Gruber "The Rationalist Myth of Hypothesis Choice."

Section 4 - Statistical Inference

* Introduction (Statistics in Scientific Thinking)

Baron, R.M., & Kenny, D.A. The moderator-mediator distinction in social psychological research: Conceptual, strategic, and statistical considerations. JPSP, 51, 1173-1182.

Zuckerman, M., et. al. To predict some of the people some of the time: In search of moderators. JPSP , 1989, 57, 279-293.

McClelland, G.H., & Judd, C.M. Statistical difficulties of detecting interactions and moderator effects. Psychological Bulletin, 1993, 114, 376-390.

Meehl, P. E. Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology (pp. 826-834 optional). Journal of Consulting and Clinical Psychology, 1978, 46, 806-834.

Schmidt, F.L. What do data really mean: Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 1992, 47, 1173-1181.

Prentice, D., & Miller, D. When small effects are impressive. Psychological Bulletin, 1992 112, 160-164.

Cohen, J. A power primer. Psychological Bulletin, 1992, 112, 155-159.

Cohen, J. Things I have learned (so far). American Psychologist, 1990, 45, 1304-1312.

Cohen, J. The earth is round (p < .05). American Psychologist, 1994, 49, 997-1003.

Leventhal, L, & Huynh, C. Directional decisions for two-tailed tests: Power, error rates, and sample size. Psychological Methods, 1996, 1, 278-292.

Rosnow, R.L., & Rosenthal, R. Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 1989, 44, 1276-1284.

Rosnow, R.L., & Rosenthal, R. "SOME THINGS YOU LEARN AREN'T SO": Cohen's paradox, Asch's paradigm, and the interpretation of interaction. Psychological Science, 1995, 6, 3-9.

Abelson, R.P. Vulnerability of contrast tests to simpler interpretations: An addendum to Rosnow and Rosenthal. Psychological Science, 1996, 7, 242-246.

Petty, R.E., Fabrigar, L.R., Wegener,D.T., & Priester, J.R. Understanding data when interactions are present or hypothesized. Psychological Science, 1996, 7, 247-252.

Rosnow, R.L., & Rosenthal, R. Contrasts and interactions redux: Five easy pieces. Psychological Science, 1996, 7, 253-256.

Carver, R.P. The case against statistical significance testing. Harvard Educational Review, 1978, 48, 378-399.

Frick, R.W. The appropriate use of null hypothesis testing. Psychological Methods, 1996, 1, 379-390.

Rosenthal, R., & Rubin, D.B. The counternull value of an effect size: A new statistic. Psychological Science, 1994 5, 329-334.

Rosnow, R.L., & Rosenthal, R. Computing contrasts, effect sizes, and counternulls on other people's published data: General procedures for research consumers. Psychological Methods, 1, 331-340.

Rosenthal, R. Science and ethics in conducting, analyzing, and reporting psychological research. Psychological Science, 1994, 5, 127-134.

Zuckerman, M., . . ., Rosenthal, R. Contemporary issues in the analysis of data: A survey of 551 psychologists. Psychological Science, 1993, 4, 49-53.

Rosenthal, R. How are we doing in soft psychology? American Psychologist, 1990, 775-777.

Rosnow, R.L., & Rosenthal, R. If you're looking at cell means, you're not looking at only the interaction (unless all main effects are zero). Psychological Bulletin, 1991, 110, 574-576.

Rosnow, R.L., & Rosenthal, R. Definition and interpretation of interaction effects. Psychological Bulletin, 1989, 105, 143-146.

EXCERTS FROM Morrison, D.E., & Henkel, R.E. (1970). THE SIGNIFICANCE TEST CONTROVERSY: A READER. Selections # 5, 17, 19, 22, 25, & 27.

Singer, M. The vitality of mythical numbers. In KST (Eds.) Judgment under Uncertainty: Heuristics and Biases. Cambridge University Press.

* Selection 26--Francis Galton "The Charms of Statistics."

* Selection 27--Paul E. Meehl "How Psychology Differs from Physics."

* Selection 28--Lee J. Cronbach "Experimental and Correlational Psychology."

* Selection 30--Michael E. Doherty, et al. "Null Hypothesis Testing, Confirmation Bias and Strong Inference."

* Selection 31--Amos Tversky and Daniel Kahneman "The Law of Small Numbers."

Section 5 - Statistical and Probabilitic Reasoning

Tversky, A., & Kahneman, D. Judgment under uncertainty: Heuristics and biases. Science, 1974, 185, 1124-1131.

Lopes, L.L. The rhetoric of irrationality. Theory andPsychology, 1991, 1, 65-82.

Gigerenzer, G. How to make cognitive illusions disappear: Beyond "heuristics and biases". European Review of Social Psychology, 1991, 2, 83-115.

Kahneman, D., & Tversky, A. On the reality of cognitive illusions: A reply to Gigerenzer's critique. Psychological Review, in press.

Fox, R. Prejudice and the unfinished mind: A new look at an old failing. Psychological Inquiriy, 1992, 3, 137-152.

Nickerson, R.S. Ambiguities and unstated assumptions in probabilistic reasoning. Psychological Bulletin, 1996, 120, 410-433.

Cohen, L. J. Can human irrationality be experimentally demonstrated? The Behavioral and Brain Sciences, 1981, 4, 317-370 (this includes 29 peer responses and the author's response).

Section 6 - The Ideology of Experimental Psychology

Goldman, A.I. The psychology of folk psychology. Behavioral and Brain Sciences, 16, 15-28.

Ring, K. Experimental social psychology: Some sober questions about some frivolous values. Journal of Experimental Social Psychology, 1967, 3, 113-123.

Sampson, E. E. Cognitive psychology as ideology. American Psychologist, 1981, 36, 730-743.

Section 7 - How "Scientific" is Psychological Science?

Planck, M. The meaning and limits of exact science. Science, 1949, 110, 319-327. Also in Scientific autobiography and other papers. New York: Philosophical Library, 1949.

Price, G. R. Science and the supernatural. Science, 122, 1955, 359-67.

(responses by Scott, Haskell, & Mundle in Nature, 1973, 245, 52-54).

Bem, D.J., & Honorton, C. Does Psi exist? Replicable evidence for an anomalous process of information transfer. Psychological Bulletin, 1994, 115, 4-18.

Hyman, R. Anomoly or artifact? Comments on Bem and Honorton. Psychological Bulletin, 1994, 115, 19-24.

Bem, D.J. Response to Hyman. Psychological Bulletin, 1994, 115, 25-27.

Newman, L.S., & Baumeister, R.F. Toward an explanation of the UFO abduction phenomenon: Hypnotic elaboration, extraterrestrial sadomasochism, and spurious memories. Psychological Inquiry, 1996 7, 99-126.

Commentaries (127-184)

Authors' Response (185-197)

Zelen, M. (1976) Astrology and statistics: A challenge. The Humanist, 36, 32-33.

Abell, G.O., Abell, A.A., Gauquelin, M., & Gauquelin, F. (1976) A test of the Gauquelin "Mars effect". The Humanist, 36, 40-45.

Gauquelin, M., & Gauquelin, F. (1977) The Zelen test of the Mars effect. The Humanist, 37, 30-35.

Zelen, M., Kurtz, P., & Abell, G. (1977) Is there a Mars effect? The Humanist, 37, 36-39.

Cronbach, L. J. Beyond the two disciplines of scientific psychology. American Psychologist, 1975, 30, 116-126.

Elms, A. C. The crisis of confidence in social psychology. American Psychologist, 1975, 30, 967-976.

Cooper, R.M. The passing of psychology. Canadian Psychology, 1982, 23(4), 264-7.

Hedges, L. V. How hard is hard science, how soft is soft science? American Psychologist, 1987, 42(2), 443-455.

Berscheid, E. Mea Culpas and lamentations: Sir Francis, Sir Isaac, and "The slow progress of soft Psychology". In The Emerging Field of Personal Relationships, 1986, 267-287.

Brush, S.G. Should the history of science be rated "X"? Science, 183, 1164-1172.

Caporael, LC. (1995). Sociality: Coordinating bodies, minds, and groups. Psycoloquy.95.6.01, 1-29.

Gergen, K. J. Social psychology as history. Journal of Personality and Social Psychology, 1973, 26(2), 309-320.

Schlenker, B. R. Social psychology and science. Journal of Personality and Social Psychology, 1974, 29(1), 1-15.

Gergen, K. J. Social psychology, science and history. Personality and Social Psychology Bulletin, 1976, 2, 373-383.

Schlenker, B. R. Social psychology and science: Another look. Personality and Social Psychology Bulletin, 1976, 2, 384-390.

Wallach, L., & Wallach, M.A. Gergen versus the mainstream: Are hypotheses in social psychology subject to empirical test? Journal of Personality and Social Psychology, 1994, 67, 233-242.

Shaller, M., Crandall, C.S., Stangor, C., & Neuberg, S.L. "What kinds of social psychology experiments are of value to perform?" Comment on Wallach and Wallach (1994). Journal of Personality and Social Psychology, 1995, 69, 611-618.


Section 8 - Ecological Validity versus Laboratory Generalizability

Banaji, M.R., & Crowder, R.G. The bankruptcy of everyday memory. American Psychologist, 1989, 44, 1185-1193.

(The following are all from American Psychologist, 1991, 46.)

Loftus, E.F. The glitter of everday memory . . . and the gold.

Conway, M.A. In defense of everday memory.

Ceci, S.J., & Bronfebrenner, U. On the demise of everday memory: "The rumors of my death are much exaggerated" (Mark Twain).

Morton, J. The bankruptcy of everday thinking.

Neisser, U. A case of misplaced nostalgia

Roediger, H.L. They read an article?: A commentary on the everday memory controversy.

Tulving, E. Memory research is not a zero-sum game.

Klatzky, R.L. Let's be friends.

Bruce, D. Mechanistic and functional explanations of memory.

Gruneberg, M.M., Morris, P.E., & Sykes, R.N. The obituary on everyday memory and its practical applications is premature.

Bahrick, H.P. A speedy recovery from bankruptcy for ecological memory research.

Aanstoos, C.M. Experimental psychology and the challenge of real life.

Banaji, M.R., & Crowder, R.G. Some everyday thoughts on ecologically valid methods.

Section 9 - The Method before Method - Creativity in the Hypothesis Generation Process

Slovic, P., & Fischhoff, B. On the psychology of experimental surprises. Journal of Experimental Psychology: HPP, 1977 3, 544-551.

Davis, M. S. That's Interesting! Towards a phenomenology of sociology and a sociology of phenomenology. Phil. Soc. Sci. I, 1971, 309-344.

Sternberg, R.J., & Gordeeva, T. The anatomy of impact: What makes an article influential. Psychological Science, 1996, 7, 69-75.

* Introduction (Scientific Creativity)

* Selection 32--Charles Darwin "On the Making of Taxonomies."

* Selection 37--Gerald Holton "On Themata in Scientific Thought."

* Selection 38--Henri Poincare "The Choice of Facts."

* Introduction (Scientific Creativity)

* Selection 39--Herbert Crovitz "An Algorithm for Creativity."

* Selection 40--Howard E. Gruber "Chance, Choice and Creativity."

* Selection 41--Mary B. Hesse "The Function of Analogies in Science."

* Selection 44--Thomas S. Kuhn "On Thought Experiments."

* Introduction (Image and Reality)

* Selection 45--Max Planck "Absolutes in Science."

* Selection 46--Gerald Holton "What, Precisely, Is 'Thinking'?"

* Selection 47--Arthur I. Miller "Visualizability as a Criterion for Scientific Acceptability."

* Selection 48--Albert Einstein "A Testimonial."

* Epilogue


*Note - Those selections designated with an "*" are from:

Tweeney, R.D., Doherty, M.E., & Mynatt, C.R. (Eds.). On Scientific Thinking. New York: Columbia University Press, 1981.

Home Page NoticesDocumentsLinksArchivesBooksTeaching Materials Administration Applying Critical Psychology Radical Psychology Journal

 Up to top

Updated 30 September 2007 - Contacts
RadPsyNet: http://www.radpsynet.org