Research settings in industrial and organizational psychology: Are findings in the field more generalizable than in the laboratory?

Abstract

The authors analyzed for content all empirical articles from 1966, 1970, and 1974 volumes of Journal of Applied Psychology, Organizational Behavior and Human Performance, and Personnel Psychology to determine types of organizations, subjects, and dependent measures studied. Contrary to common belief that field settings provide for more generalisation of research findings than settings do, field research appeared as narrow as research in actors, settings, and behaviors sampled. Indeed, industrial-organisational psychology seems to be developing in a psychology of college student, and in field, a psychology of self-report of male, professional, technical, and managerial employees in productiveeconomic organizations. The authors suggest that coordinated strategies of research in both and field settings are needed to construct an externally valid industrial and organizational psychology. Should industrial and organizational psychologists conduct their research in settings? For many industrial and organizational psychologists, answer to this question would be an emphatic no, but others would defend as a legitimate setting for research (Fromkin & Streufert, 1976; Weick, 196S). The relative merits of and field settings have been debated within many areas of psychology, including comparative psychology (Miller, 1977), environmental psychology (Proshansky, 1976), social psychology (Ellsworth, 1977; McGuire, 1967; Ring, 1967), race relations (Fromkin & Ostrom, 1974), and perception (Gibson, 1966). Two common criticisms of research are that settings are susceptible to experimental artifacts (Rosenthai & Rosnow, 1969; Silverman, 1977) and .serious ethical problems (Kelman, 1967). Although these two criticisms have been debated in numerous articles and have not been resolved entirely, Fromkin and Streufert have argued persuasively that artifacts and ethical problems also exist in field settings and are not unique to laboratory. A third criticism of research is that artificiality of imposes severe restraints on external validity of findings. Even those critics who concede that settings provide for more elimination of alternative interpretations of results than typical field settings do are often doubtful that one can generalize findings from to predict, understand, or control real-world phenomena. A perusal of recent Handbook of Industrial and Organizational Psychology (Dunnette, 1976) reveals differences of opinion that exist among industrial and organizational psychologists with regard to generalizability of research. Ohapanis (1976) observed that most experiments in psychology have only very limited relevance for solution of practical Vol. 34, No. 2, 141-150 Portions of this article were presented at meeting of Midwestern Psychological Association, Chicago, May 1977. The authors' names are listed in alphabetical order, reflecting equal contributions by both to design, implementation, analysis, and writing of this study. The authors thank John Dzamba, Claude Mattox, Brian Robinson, and William Wratten for their assistance with data analysis. Requests for reprints should be sent to Robert L. Dipboye, Rice University, Department of Psychology, Houston, Texas 77001. AMERICAN PSYCHOLOGIST • FEBRUARY 1979 • 141 Copyright 1979 by American Psychological Association, Inc. 0003-066X/79/3402-0141$00.75 problems. . . . Unsuspected interactions in real life may nullify or even reverse conclusions reached in (p. 730). Similarly, Bouchard (1976) observed that laboratory experiments seldom deal adequately with boundary conditions or context factors and therefore, lend themselves to unjustified and often erroneous extrapolations (p. 364). In contrast with these arguments, Fromkin and Streufert (1976) argued that the artificiality of laboratories is being unjustly elevated to status of a fatal flaw (p. 433). Furthermore, they asserted that instead of reinforcing myth that settings seldom yield data which are relevant to real world problems, it is proposed that settings merely impose identifiable limitations upon range of criterion situations to which a particular set of findings may be practically applied, (p. 442) Despite firm convictions held by proponents and critics, many of arguments for and against external validity of research are based on stereotypes rather than data. One such stereotype has been that a field setting, because it is natural to subject and not contrived, automatically provides for more generalization of results than an artificial setting does. However, problem of external validity is one of making inferences not only from settings but from actors and behaviors of these actors (Runkel & McGrath, 1972). Critics of research often place inordinate weight on setting, in discussions of external validity, to neglect of actors and their behavior. For instance, some have assumed that because setting in field research is typically an actual organization, data collected in such a setting must be more generalizable to other organizations than are data collected in laboratory. However, there are differences among organizational settings that are as large as differences that exist between an organizational setting and a laboratory. Rather than assuming that field research in abstract is more generalizable, there is a need to determine types of organizations with which field researchers have been concerned. The belief that research in industrial and organizational psychology is less externally valid than field research also results from tendency to infer from setting characteristics of actors and their behaviors. Such inferences are often without empirical support. For example, assumption has been made that field research involves subjects who are more representative of working population than college students typically used in laboratory. But is this assumption correct? Instead of rejecting research on basis of an untested stereotype, we need to examine more carefully who participants are in and field research. Another assumption has been that research in field settings has as focus of investigation natural behavior, defined by Tunnell (1977) as behavior that is not established or maintained for sole or primary purpose of conducting research; behavior is part of person's existing response repertoire. Whether procedure used to record behavior is human or mechanical (e.g., videotapes, audiotapes, explicit behavioral checklists) essential naturalness of observation is maintained. . . . Self-reports should not be considered natural behaviors unless they are made by person in real life. (pp. 426-427) The critics of research often seem to assume that dependent variables examined in field are more natural and thus more externally valid than dependent variables examined in laboratory. But what are dependent variables used in and field research? Again, arguments for and against external validity of research have been based on assumptions rather than data. The issue of relative external validity of and field settings is an important one not only to industrial and organizational psychologists but to all psychologists conducting and field research or utilizing findings of such research. A more objective analysis of issue of external validity is needed than has appeared in previous literature. In this article, our intent was to test empirically some of assumptions underlying belief that findings from field research are more generalizable to other settings, behaviors, and actors than are findings of research. Of course, one cannot conclude that any one study or set of studies possesses complete external validity (Campbell & Stanley, 1963, p. 17), but one can discuss relative limits that seem to exist for generalizing findings from research using different strategies. The intent of this article is to assess empirically limits that seem to exist for recent and field research in industrial and organizational psychology. In order to achieve this objective, articles in three major journals were analyzed for content to determine types of organizations, persons, and behaviors studied. 142 • FEBRUARY 1979 • AMERICAN PSYCHOLOGIST