The test procedures provided data for use in the further improvement of the system and also provided a basis for measuring the validity, reliability, and practicality of the system. The degrees of system validity and reliability are reflected in the coefficients of correlation computed and set forth below. The observations and comments from panel participants provide a measure of the system's adequacy as a viable tool for use by management in evaluating jobs. In making the analyses of the tests, the consensus whole job ranking by each panel was used as the basic criterion against which to compare other decisions and evaluations by that panel. (As a check on the validity of treating the panel whole job ranking as the criterion, the rank order correlations were computed between each such panel ranking and available BLS pay relative data. These coefficients of correlations were .964, .950, and .987 for agencies A, B, and C respectively. The similar coefficient of correlation for agency D was .400 due in substantial part to the skewed nature of the agency sample jobs--see the footnote to the following table--and to the fact that pay relative data was available for too few of the positions.) The results of some of the comparative analyses are shown in the following table: Comparison "1" in the above table indicates the degree of agreement or disagreement by each panel with the present General Schedule alignment of the test jobs. Comparison "2" in the above table reflects the degree of agreement or disagreement of the point ratings by each panel with its own whole job ranking. This comparison rather clearly shows that the proposed COMOT ratings are more like the panels' whole job rankings than are the current General Schedule grades of the test jobs. Comparison "3" demonstrates further that the panel point ratings are more like the whole job rankings than they are like the current General Schedule grades of the test jobs. This point is further confirmed by Comparison "4" in the above table. Comparison "5" confirms the internal consistency of the COMOT skill levels with the panel point ratings. Overall, these test results indicate that the system has a relative high degree of validity. * Agency D included in its sample of test jobs some positions on which there was extensive disagreement on the present grade. The test jobs selected by this agency also covered a narrower range of job levels than did the test jobs selected by the other agencies. The skewed nature of this selection of jobs produced lower coefficients of correlation than were observed in the other agencies. consistent results from one rater to another. The test procedures were designed to produce data comparing the actions of each individual panel member with the consensus whole job ranking, thus providing a measure of reliability. These results were produced by panelists who receive no specific training other than that provided in a simple instruction less than two pages long. Five of the 20 panelists were supervisors with little or no previous job evaluation experience. The job descriptions used were written in the standard agency format which always differed markedly from the factor and benchmark position format. All of the panelists continued their regular duties in addition to finding time in their busy schedules to do the ratings. Because the COMOT system was still in a developmental stage, the panels were furnished with benchmark descriptions that in most cases were not directly applicable to the positions assigned to the panels for rating. The adverse conditions under which the test was conducted emphasize the significance of the relatively high degree of correlation between the individual ratings and the panel consensus. These data indicate that the fully developed system will produce reliable results. Pay Relationships under the COMOT System As was mentioned earlier in the section headed Testing the Soundness of Job Relationships, the General Schedule does not produce inter-job relationships that correlate well with salary patterns used by other employers. Therefore, an analysis of the job alignment provided by the COMOT system and its correlation with other salary patterns is especially important. Twenty-five of the benchmark positions in the most recent version of the COMOT system are directly represented in the BLS National Survey of Professional, Administrative, Technical, and Clerical Pay. A simple arithmetic average can be computed for each General Schedule grade and each COMOT skill level using the same 25 positions. The extent to which the average salary of each of the positions differs from the GS grade or COMOT skill level average provides a simple measure of the relative relationships. The extent of this deviation is shown in the following table: Deviations of 25 Individual Job Salaries from General Schedule Grade These data indicate the COMOT system can provide job evaluations and salary schedules substantially more comparable to private industry practice than does the General Schedule. Opinions Expressed by the Panelists Participating in the Test Of equal importance in evaluating these test results are the observations and comments of the agency panelists. The nature of the comments were fairly consistent and favorable among the four panelists. Some of these were: --The present job descriptions are inadequate; both content and format can be improved. A part of the present inadequacies in job descriptions arises out of a lack of understanding by supervisors and employees as to what is needed. Supervisors work in a blind alley in the present system. These problems will be greatly alleviated by the COMOT system. --The COMOT system needs more benchmark job descriptions more specifically directed at agency jobs. --The COMOT system will blend in well with present agency benchmark jobs and standard job descriptions. --The COMOT system will be better understood by supervisors. --The COMOT system is much better than the present standards system. Job evaluators can become more proficient quicker and produce more reliable results. --The COMOT system should help improve supervisor-employee relations. The system is easy to understand. There should be fewer arguments since there is less to argue about. --The rating system requires that all jobs be broken down and evaluated on the same four factors which is an advantage over the present system. It also acts as a common denominator. --The rating plan appears to be understandable, easy to apply, and equitable. --The idea of placing classification responsibilities with the supervisor is good and should result in more realistic grade structures, and a greater understanding and use of classification by management. --Administration should be delegated to the lowest possible management level. Use of a committee composed of supervisors, position classifiers and possibly union representatives should be considered. Improvements Made on the Basis of the Test Subsequent to the four agency tests the factor scales were editorially refined to minimize the difficulties that the panels encountered. Additional benchmark positions were added to broaden the occupational coverage of the tentative COMOT system. The benchmark positions were all reviewed in depth to assure the internal consistency of the system. The various indices of the benchmark positions were rearranged to improve their usability. The revised COMOT system was then submitted to all agencies and to the interested employee organizations for study, trial application, and comment. The COMOT system includes all occupations in which the principal nonsupervisory positions at the full performance level are: --of a clerical or office machine operator nature, or are those of technicians performing work subordinate to and supportive of professional, administrative, or technological work; and --like nonexempt jobs as defined in the Fair Labor Standards Act of 1938; and --like jobs typically paid on a locality basis by other employers; and --most typically are filled by persons with less experience and education than is generally considered to be equivalent to a baccalaureate degree based on four academic years of study. The system specifically includes the following occupations: (This list will be developed and issued by the Civil Service Commission prior to installation of the COMOT system, and will be revised thereafter as necessary to assure its currency at all times.) 364 for CLERICAL, OFFICE MACHINE OPERATION, AND TECHNICIAN POSITIONS This is a factor ranking type of evaluation system using both factor rating scales or guide charts and benchmark position descriptions. It provides seven skill levels, exclusive of purely trainee positions. Trainee positions under this system are not classified into skill levels, but rather are paid a percentage of the salary for the skill level of the lowest productive position in the occupation for which the employee is being trained. How to Evaluate Positions The first step in the evaluation of a position is to compare the position with the appropriate benchmark descriptions. In most instances, a benchmark position that is essentially identical with the position to be evaluated will have previously been approved. In such case the evaluation of the benchmark position is also the appropriate evaluation for the new position. In the event the benchmark position is similar to, but not precisely identical with, the position to be evaluated, the rating scales or guide charts can be used to assess the significance of the observed differences. The guide charts can also be used in accordance with applicable regulations and delegations of authority to evaluate positions for which no appropriate benchmarks have been previously approved. Using the Rating Scales When, in accordance with delegated authority, the rating scales or guide charts are used to evaluate a position the first step is to compare the position with the rating scales, (using the benchmark positions to define and clarify the scales) and identify the level which best fits the job being rated, and record the point score for that step. No point scores may be assigned other than those on each factor chart. The reason for this is that the point value assigned to each step on each factor scale represents a range of jobs. Note, for example, that each Index of Benchmark Positions by Factor shows a number and variety of positions for individual steps. Since the point values provided on the several factor scales encompass all of the possible job levels that are within the scope of the total scale, no intermediate point values can be interpolated. Every job should have a point score on each factor. The point scores are recorded and the factor score totaled. This total score represents the relative value of this position and is used to assign individual positions to levels by applying the conversion table. Benchmark positions have not as yet been provided for each score in each factor. The absence of a benchmark position to illustrate a specific step does not restrict the use of that level of the factor rating scale. Eventually, all levels will be backed up with appropriate benchmarks. Mixed Duty Positions In point scoring a position which involves the performance of work in two or more occupations, or more than one level in the same occupation, each factor is evaluated on the basis of a study of the individual duties. The evaluator should take into account duties that are significant in terms of the overall purpose of the position and are performed on a recurring basis. Normally, the highest level significant duty controls the determination of the factor score. (It is expected, however, that specifically applicable benchmark descriptions will be available for most mixed duty positions, and such benchmarks will be the principal means for evaluating such positions.) |