Chapter 31. STANDARDS FOR STATISTICAL METHODOLOGY Background Standards for statistical methodology should cover the design, collection, processing, tabulation, analysis, evaluation, and publication of Federal statistical data. The general principles governing these standards are described in the Statistical Policy Handbook published by the Office of Federal Statistical Policy and Standards (OFSPS). The implementation of these principles and their practical applications must take account of the different types of data collected and the purposes which the data are intended to serve. Thus the precise standards of quality which are appropriate for specific bodies of data must be determined by the Federal agency which sponsors the collection of these data; however, interagency interest must be taken into account. In addition, each sponsoring agency, as well as the collection agency in the case of interagency agreements, should be responsible for establishing and maintaining an assessment and monitoring program embracing each of the above phases and covering both general-purpose and special-purpose statistics, in order to ensure that the prescribed standards are followed and that problems which arise can be identified and addressed within the agency before the data in question are released to the public. This assessment program must include a review of the methodology employed and the underlying concepts and definitions. Standards for Statistical Methodology at Present Published OFSPS Standards The Statistical Policy Handbook has as its purpose "... to establish uniform statistical standards and guidelines for the collection and compilation of statistical data and for the release and publication of Federal statistics." The Handbook states that "... the standards and guidelines prescribed by the Directives are to be followed by agencies in the executive branch of the Federal Government and by federally sponsored contractors and grantees in the direct collection, compilation, and publication of statistical data." This follows the legislative philosophy of Section 103 of the Budget and Accounting Procedures Act of 1950 which states that the established statistical standards shall be followed by the agencies. The following Directives of the Statistical Policy Handbook give standards relevant to statistical methodology: -Directive No. 1-Standards for Statistical -Directive No. 2-Standards for the -Directive No. 11-Standard Data Source for -Directive No. 13-Standard Data Source of -Directive No. 15-Race and Ethnic Standards Directive No. 1 of the Handbook, Standards for Statistical Surveys, establishes the need for a clear purpose of the survey, as well as the development of the various aspects of the survey plan, the questionnaire and the preparation of the final report. Among the standards established are (1) that samples of the universe should be probability samples and (2) that "...if probability sampling is used, sampling error should be estimated...." Nonsampling errors should also be considered in establishing proper controls of the survey operations and evaluated to the extent possible. Directive No. 2, Standards for the Publication of Statistics, states that "...full information should be available to users about sources, definitions, and methods used in collecting and compiling statistics, and their limitations." This latter standard establishes that "...there should be available to the user an appraisal of the statistics which would make it possible to evaluate their appropriateness for any intended use." Directive No. 11, on statistical estimates of labor force and unemployment, designates official estimates for these statistics. Directive No. 13 designates specific data series on population prepared by the Bureau of the Census to be used for distributing Federal benefits. The importance of this standard is that the selected official series must be used by any program which uses population statistics in distributing Federal funds. The methodology used to derive these official estimates thus becomes very important. Directive No. 15 provides standard classifications on race race and ethnicity for use in Federal administrative reporting and statistical activities. Examples of Individual Agency Standards In March 1974, the Bureau of the Census issued Technical Paper 32 (TP 32), Standards for Discussion and Presentation of Errors in Data. A memorandum from the Director of the Bureau of the Census to all Divisions and offices when TP 32 was published stated that "...this publication, the first of its kind, formalizes and extends practices which the Bureau has for some years followed in recognition of its responsibility for informing users of important limitations in its data. ...Flexibility in applying the standards will be needed to take account of different forms of publication, including press releases and distribution of unpublished data in either computer readable or hard copy form, and of limitations of our knowledge and techniques...." These standards for presentation of errors were used as background material for six seminars held in 1974 under the sponsorship of the Statistical Policy Division of OMB to discuss the responsibility of producers of statistics to inform the users of the limitations of the data published. The standards cover both sampling errors and nonsampling errors and recognize that the total error is larger than the published sampling errors. Although these standards are not official standards, they can be useful as guidelines to agencies preparing statistical reports in informing users on the limitations of the data. The National Center for Health Statistics (NCHS) has prepared a Manual on Standards and Procedures for Reviewing Statistical Reports, which is used internally for reviewing statistical statements in reports on the Vital and Health Statistics series. The material in this manual serves as a guide to NCHS personnel involved in reviewing statistical text statements. Another section of the manual gives examples of sampling errors for specific NCHS surveys and procedures for calculating sampling errors of certain derived statistics. In addition to formalized standards on statistical methodology, agencies have self-imposed standards which influence the quality of the data. For example, any survey done by the Bureau of the Census for another Federal agency will be carried out in accordance with the Census Bureau's standards and expertise. The same is true for surveys conducted for other agencies by the statistics unit of the Economics, Statistics, and Cooperatives Service of the Department of Agriculture. Another example is the standards which the NCHS imposes on the States for the collection of data on vital registration (births and deaths) and other information obtained through the Cooperative Health Statistics System. These standards are not applied uniformly by all States. (This problem is discussed in greater detail in the chapter on FederalState-local cooperative systems of data collection.) Office of Federal Statistical Policy and Standards Functions The three functions of the OFSPS which relate directly to the establishment and maintenance of standards for statistical methodology are clearance of statistical surveys, statistical policy coordination, and legislative review. OFSPS is responsible for reviewing and clearing any statistical survey which falls within the purview of the Federal Reports Act of 1942 as amended. In reviewing these proposed projects, OFSPS follows the standards for statistical methodology which have been established; therefore, the clearance function of OFSPS is one means whereby the application of the standards can be monitored. In carrying out its statistical policy coordination function, OFSPS can, in addition, inform particular agencies of the standards to be followed in conducting particular surveys or statistical operations and can provide for an exchange of relevant experience and information. Finally, in reviewing proposed legislation in terms of its implications for statistical data requirements, OFSPS plays an important role in determining the specific data requirements, the best means for collecting these data, and the appropriate formulas to be employed in deriving the particular measures called for in the legislation. Federal Committee on Statistical A major current effort to develop improved standards for statistical methodology is the work of the Federal Committee on Statistical Methodology (FCSM), chaired by the Office of Federal Statistical Policy and Standards. The members are selected because of their recognized expertise in the area of statistical methodology. The FCSM has formed subcommittees to investigate specific areas. During 1976, five subcommittees of the FCSM were established to address the following selected areas of concern: 1. Statistics for allocation of Federal funds to local areas. The study of the programs which allocate Federal funds to local areas is designed to determine the appropriate guidelines to be followed in prescribing the statistics and formulas to be used in allocating program funds. An additional topic being investigated by this subcommittee is the possible effect of statistical adjustments of biased data in improving the allocation of funds. A Report on Statistics for Allocation of Funds prepared by this subcommittee was published as Statistical Policy Working Paper 1 by OFSPS. This report includes recommendations concerning the data and formulas used for the distribution of Federal funds to State and local areas. 2. Problems of data disclosure.-This study aims at suggesting guidelines to be followed by agencies in releasing data in the form of statistical tables and microdata files without violating confidentiality restrictions. A Report on Statistical Disclosure and DisclosureAvoidance Techniques prepared by this subcommittee was published as Statistical Policy Working Paper 2 by OFSPS. This report includes recommendations on disclosureavoidance techniques in tabulations and microdata files, as well as the effects of disclosure on data subjects and users. 3. Matching data records.-In this study, alternative methods of matching data files from different sources are being investigated from the point of view of confidentiality requirements. The study is focused on matching techniques for data from different Federal agencies for which access at the individual record level is restricted. 4. Nonsampling errors.-This study seeks to assemble illustrative information concerning sources and types of nonsampling error affecting a number of key Federal statistical programs. The study developed an "error profile" for employment statistics derived from the Current Population Survey. Additional error profiles based on selected Federal statistical series would illustrate further this approach. A glossary of terms related to nonsampling errors is also being prepared. 5. Standards for Federal-State-local cooperative systems of data collection.-This study addresses the development of appropriate standards for these cooperative data collection systems. This topic is discussed in a separate chapter of the Framework. During 1977, two new subcommittees were established to address the following topics: 1. Contracting statistical surveys.-The process by which Federal contracts are established and monitored will be reviewed. The goal is to establish guidelines that would improve the quality of statistical data obtained through 2. contracts. Statistical uses of administrative records.-Uses of administrative records for statistical purposes will be investigated to determine advantages and limitations of such procedures. The work of the FCSM and its subcommittees should serve as a focal point for the development of standards for Federal statistics. The promulgation of the standards is the continuing responsibility of OFSPS in its coordinating and policy role. The implementation of these standards, however, must remain the responsibility of the particular agencies which produce the statistics in question. In addition, agencies which sponsor statistical data collection efforts must monitor the design and implementation of these efforts to assure adequate quality of data produced. Standards for Statistical Surveys The problem.-Statistical surveys, whether complete censuses or sample surveys of some universe of interest, are used to obtain many of the statistics needed by Federal agencies. These surveys vary in quality. In order to ensure that the statistics collected fulfill the purposes which they are intended to serve, a clear statement of the objectives of the data collection effort is required, together with an adequate survey design. It is necessary to investigate the questionnaire design, sample design, interviewing strategies and procedures, processing procedures, and so forth. to ensure that the survey design is adequate. In some cases, a statistical survey may not be required to collect the necessary information. For example, the development of new methodologies or the exploration of previously unexamined problem areas may be carried out more effectively by means of a case study approach or by means of small-scale pilot studies. Alternatively, the information may be available from administrative records. In this case, the agency wanting to use the records for statistical purposes and the agency administering the records need to coordinate the transfer of the data. Policy recommendations.—In addition to submitting proposals to OMB and the Office of Federal Statistical Policy and Standards (OFSPS) after surveys have been designed, it is recommended that clearance requests be submitted for review to the OMB and OFSPS at an early stage, especially for projects of major importance (e.g., projects with budgets over $500,000 or projects contacting very large samples). The review of proposed statistical data collection efforts at an early stage could result in the identification of weaknesses at a time when they could be eliminated at minimum cost. This would also lead to improved coordination of statistical data collection systems, and to better methodologies and adherence to standards. Moreover, the early review of projects would greatly expedite final clearance by having most major issues resolved at the beginning of the project. In addition, it is recommended that pretests be a mandatory requirement for new projects with budgets over some predetermined amount. The size of the pretest should be commensurate with the size of the proposed new survey. Guidelines for contracting for statistical surveys should be developed. These guidelines would be used by the agencies in issuing any request for proposal (RFP). The guidelines should cover the planning, design, implementation, and monitoring of the survey. The Standards for Statistical Surveys (Directive No.1, Statistical Policy Handbook) has been Statistical expertise is essential when an office is designing a statistical survey. Not all offices have adequate statistical expertise on their staff, so that it would be necessary to make available to these offices the required resources for an adequate design of new statistical projects of major importance. For statistical projects of general character, the statistical expertise of the Bureau of the Census might be the best resource; for specialized surveys, it would be more appropriate to consult with agencies with specific expertise such as the National Center for Health Statistics, the Bureau of Labor Statistics, the Office of Research and Statistics of the Social Security Administration, the National Center for Education Statistics, the Economics, Statistics, and Cooperatives Service of the Department of Agriculture, and so on. Feasibility Study of a Centralized Clearance System for Publication of Major New Series of Data The problem.-Differing standards are being applied by the various Federal statistical agencies with regard to the quality of the data being released, and with respect to the information given to users on the quality of the data released. It is recognized that agencies producing Federal statistics have a responsibility to users of the data. Data released to the general public should be free from serious shortcomings; in addition, statistical publications should include a discussion of the limitations of the data. (A related set of recommendations is contained in the chapter on user access to Federal, public data files.) Policy recommendations.-The feasibility of instituting a centralized clearance system for initial publication of data from new Federal statistical series should be studied by a committee with members from various agencies. Under such a system, initial issues of publications of data resulting from new Federal statistical series would be sent to the OFSPS for technical review before their release. This centralized |