science fails to keep pace. In these fields of science, we should take special precautions to insure that applications remain rooted in and guided by basic, disciplined research.

We should be hesitant about asking for evaluation research on Federal programs unless we are prepared to support the prior research into human behavior, social organization, and the political process. We should be wary of the widespread application of social science methods in government and industry, unless we support basic research in statistics, measurement, and observation.

To reduce the support for basic social and behavioral science at NSF will not seriously slow the growth of social R. & D. On the contrary, it might make things worse. It will allow the growth to proceed without insuring the refinement and replenishment of basic skills, concepts, theories, and models in the social science disciplines. Accordingly, I urge you to protect the already limited budget for basic research in these fields.

I would like to add a few thoughts to my prepared comments, Congressman Brown, somewhat in response to the question you raised earlier this morning. I'd like to do so by telling a story.

The Social Science Research Council was recently approached by a mission agency of the Federal Government asking up to manage, take on a major new research endeavor with a rather sizable budget attached to it.

We are in the process of trying to explain to that agency that we do not think the basic research has yet been done to merit the spending of the magnitude of dollars that that agency is talking about. Moreover, even as we begin to share with them our own concerns about how to prepare a somewhat more modest, more focused, and we think more intelligent research project, I am nervous about whether I'm going to be able to find the talent to conduct the innovative research that would be called for in this area.

So, it's not only that we are under enormous pressure to use our skills and to apply our methods, but in many instances, we are under pressure where we are not yet prepared to specify the behavior, to apply the tools that are going to be required. We don't have the numbers of people that are necessary with the sort of deep social science expertise that is called for. It really is the National Science Foundation, and almost only the National Science Foundation, which continues to appreciate the need for replenishment, not only of the skills and the designs and the methodologies, but also the supply of personnel who are committed to deepening our understanding, to making somewhat more practical our judgment.

Therefore, I really do emphasize that the very limited budget for basic research in the social sciences that comes under the management of the National Science Foundation plays a quite critical and key role in trying to link this enormous growth of social R. & D. back to some of the basic work that is going to have to give it direction and give it discipline.

Thank you.

Mr. BROWN. Well, I want to thank you, Dr. Prewitt. That was a very well-done statement. Your bottom line, a very modest, you might even say conservative request for merely protecting this limited budget is somewhat unusual.

Dr. PREWITT. I read the hearings last year, sir. [Laughter.]

Mr. BROWN. In regard to the addendum that you made with regard to the research proposal for which, really, the basic science wasn't there yet, which you felt that the Research Council shouldn't undertake for that reason, do you feel that other possible contractors would have the same scruples?

Dr. PREWITT. No, sir. I have been told that in all likelihood, because of our refusal, there will be an RFP released and there is no doubt that that project will be bid and conducted. May I say that the magnitude of dollars is something like $5 to $6 million, which is larger than the total, for example, political science budget in the National Science Foundation.

Mr. BROWN. It's unfortunate that that kind of proposal cannot be more adequately coordinated with the status of the science and with the focus of perhaps remedying some of the basic research needs at the same time.

Dr. Mosteller, I'd like to have you go ahead at this point. I say that even though I would like to ask some additional questions of Dr. Prewitt, but we have to vacate this room in about 20 minutes. I want to make sure that you have ample time to present your statement.

Dr. MOSTELLER. Thank you, sir. Although I am president of the American Statistical Association, I appear here as a private citizen, of course, and also as one whose research in theory and applications in statistics is partly supported by the National Science Foundation.

The testimony earlier has often been broad, and I concentrate on social science as a developer of methods. I regard this aspect as very important because having the tools to find something out is as valuable, and sometimes more valuable, than what is found out itself.

With your permission, I'll speak of recent applications of new methods of social science research and then about some even newer methods and finally about some social science research that I think the country needs.

Perhaps one of the most important advances of social science research in the past decade has been the application of research methods to the evaluation of important social programs. Beginning with the design of the New Jersey income maintenance studies in the late 1960's, there have been a series of major intervention studies sponsored by the Federal Government which have involved social scientists and statisticians.

This work has been initiated in the belief that systematic experimental trials of proposed social programs have valuable advantages over other ways of learning what programs are effective, under what circumstances, and at what cost.

I might interpolate that at one time, economists felt that the experimental method was not a possibility for their field, but we have now a few strong economic studies using experimentation.

As I mentioned to you this morning, George Box, a famous statistician, said if you want to understand how a complicated system changes when you change, there is just one way to find out and that's to change it and observe what happens.

These national programs have included the experimental housing allowance program, the national health insurance study, and the Seattle-Denver income maintenance experiment. On a similar but

smaller scale, social scientists have been involved in a wide range of experimental programs in the fields of education, communications, criminal justice, mental health, and manpower training.

Knowledge derived from such efforts does not come easily. Transferring experimental technology from the lab into field settings that often include hundreds or thousands of individual introduces major methodological, to say nothing of political, difficulties.

There is a fundamental tension between the desire for a research design which provides a fair and strong test of the relevant hypotheses and the simultaneous need to plan evaluations that are financially feasible and cost effective.

For example, an important technical fact about the size of an experiment is that the "square root" law prevails. That is, doubling the number of participants in a study will not halve the unreliability of results. One must quadruple the size of a sample to halve the unreliability.

Although statisticians and social scientists are helpful in evaluating social programs when it comes to analyzing data and interpreting findings, their more important contribution comes in determining the initial design of experimental studies.

groups

A major task in evaluation research is the careful selection of of individuals to participate in the investigation, a sample that will allow researchers and policymakers to generalize results to future recipients of the program.

New methods, some mathematical, some statistical, and some just plain social science, have had to be developed for this task. In the New Jersey income maintenance experiment, economists Harold Watts and John Conlist developed a model for the selection of program participants which took into account anticipated labor market response, and they incorporated such factors as the structure of local employment opportunities. The development of such sample selection models requires both sophisticated statistical and mathematical skills and substantive knowledge of social phenomena relevant to the program being evaluated.

Yet, the cost of large-scale research efforts and the difficulty of determining appropriate samples are only two of the many problems inherent to the assessment of social interventions. The implementation of social programs can be a complex process requiring the delivery of services to different groups of individuals in different geographic locations.

As the design for such social programs becomes more complex, the tasks required to evaluate the programs become vastly more difficult. For example, a host of problems stem from the fact that the target groups for some social programs are highly mobile, both in terms of residence and occupation.

It is time-consuming and costly to deliver social services to clients who change location frequently or whose first language may not be English. As difficult as it is to deliver services, it is even more difficult to obtain and maintain accurate records on program recipients and relevant control populations.

Incomplete information and attrition are common in such studies. These problems are amplified when one wishes to assess the longrun impact of a social program.

I have mentioned these difficulties to underscore the fact that knowledge to inform important social policies does not come easily or cheaply; but I am here to tell you that we are making progress, and that significant methodological advances have been made in recent years. Social scientists and statisticians, often working together, have developed mathematical and computation methods and applied a growing range of new design and analytical techniques for studies. This work has benefited from similar efforts in the fields of biology and medicine, and from basic research and methods development in the social sciences more generally. Principles of analysis developed in studies such as the national halothane study, which assessed the safety of various anesthetics employed in surgery, can be applied to evaluating the impact of various social programs.

My colleagues and I, working on this study, the national halothane study, employed a variety of new analytical approaches, and as a result of that work, we realized that the social sciences needed a substantial body of statistics in this area and our project produced a book on what are called log-linear methods so that the techniques are now widely available.

These techniques are currently being utilized and adapted to the analysis of experimental data from the intervention studies I have mentioned. In addition, just as results from the halothane study revealed the importance of identifying and understanding interhospital differences, so programs such as the income maintenance experiments must come to understand how payment programs may operate differently in different geographical locations.

I should like now to mention some of the newer techniques which are just coming into wide usage. They are general methods for use in all Science, not just for program evaluation. Every substantial set of data has special quirks in it and one problem for the data analyst is to get the data to reveal what is basic about the set and to skip over what is only a wild or unusual value.

We have, over the last decade, developed systematic methods for what is called exploratory data analysis. These methods tend to be flexible, resistant, and robust. They are flexible because they can tackle all sorts of data and be responsive to many kinds of questions, resistant because they are not much affected by a few wild values, and robust because they get nearly all the information out of the data. even when the usual theoretical asumptions fail.

The National Science Foundation has supported the development and exposition of these methods. To obtain the robustness-efficiencyproperties, we have to use methods that require special high-speed computations. I am pleased to report that David Hoaglin and Paul Velleman have just submitted to the printer a volume of special computer programs to assist in making these important new methods. widely available.

Looking now in another direction, I have for some time been concerned about the improvement of statistics for public policy. Both legislators and agency heads often say that they need better statistics than they are being supplied in order to carry out their business properly.

They understand, of course, that brand new questions often require brand new data. What they wish are data more relevant to their

problems once the problem has been identified. These more relevant data are to help with the design of legislation or with the execution of the program.

This complaint about lack of relevance arises so often that I have concluded that what is needed in government statistics is a sequence of studies that joins the statistical experts with legislators and agency heads through social science researchers so that we can learn more clearly what should be provided.

I note that policymakers are not in this instance asking for better or more accurate statistics, but rather more relevant statistics. We need to study this matter and see whether it is possible to provide more relevant numbers. To do this will require us to see how and when statistics are used by our policymakers. Finding this out might well be a profitable cooperative venture that will require its own research methods and may lead to considerably improved government statistics. Now to summarize, National Science Foundation funding has been most important in developing new research methods for social science and economics and it will continue to be important as we try to improve their ability to contribute in the future.

Thank you. Let me express again my appreciation for this opportunity to testify.

Mr. BROWN. Thank you very much, Dr. Mosteller.

Both of you gentlemen touched upon areas which seem to me to hold a great deal of significance for the future of social science research. You mentioned the boiler investigation, Dr. Prewitt, and you focused a little bit on evaluations research, Dr. Mosteller.

The boiler case is very well known around here and it is sometimes given as the first example of a technology assessment program. One of the things that we are learning as we move rather tentatively into technology assessment, which we have been doing now for 5 or 6 years, is that successful technology assessment can't be done without a lot of social science research. That is, it involves understanding impacts on human beings of various results of a hypothesized technological development.

It means you have to create scenarios and then evaluate them extensively in terms of human reaction. I don't think we have developed a systematic theory for doing that. It seems to me to be an area in which we probably ought to.

Likewise, the evaluation research that you mentioned has sort of crept up on us, as you say, in the last few years. I suspect that we haven't yet appreciated the extent to which this can help us in a policy way. Essentially, almost everything we do could be treated as a research project and could be evaluated systematically and yet, we don't.

Take the problem of political alienation, for example. I think it's highly likely that we are creating a system which, in itself, generates political alienation and yet, we don't have the systematic analytical social science studies to determine that.

I blame all of you for that. [Laughter.]

Mr. BROWN. But, to do that kind of broad-gaged research, you have to look at the social science research needs in a rather broad framework. I sense that this is one of the problems that we have within the science. I can understand the reasons for it. I'm not try