Interrater Reliability of Joule - A Functional Capacity Evaluation System
Joule Home Page Or Back to Articles

Interrater Reliability of Joule

An FCE System by Valpar

May 1999

Ó 1999- Valpar International Corporation




Introduction

Joule is a new functional capacity evaluation system (FCE) by Valpar International Corporation. "(FCEs) are supposed to define an individualís functional abilities or limitations in the context of safe, productive work tasks" (King, Tuckwell, & Barrett, 1998). This report will not describe FCEs generally; readers unfamiliar with FCEs should refer to the article by King, Tuckwell, and Barrett (1998). That article not only described existing FCE systems, it also calls attention to several shortcomings among those systems and makes recommendations for the development of well-designed FCEs. A recent article published by Valpar International Corporation (Christopherson & Ruprecht, 1999), introduced Joule, described several of its unique features, and made a preliminary report on Jouleís effectiveness and client satisfaction. The article previewed several planned studies that would assess Jouleís interrater reliability and validity, and this article reports on the interrater reliability study.

Those who desire more detail on the specifics of Joule, should refer to the article by Christopherson and Ruprecht (1999) or contact Valpar.

Several key elements of Joule were analyzed for the purposes of this study. All of the data was gathered from the two Joule forms reproduced in Appendix A of this report, "Summary of Functional Abilities," and " FCE Participation Index." Those elements are:

This report will discuss each of the several analyzed elements of Joule separately, and will describe the manner in which each elementís " agreement index" was calculated.


Method

The study took place at two clinics: Rehabilitation Services, Inc. (RSI) in Arlington Heights, Illinois; and Nova Care Airport Medical Clinic in Bloomington, Minnesota (Nova Care). Data was gathered over a period of several days at each clinic during December of 1998 in the case of RSI, and in March of this year in the case of Nova Care.

At both clinics two trained therapists were joined by their trainer, Mary Ruprecht, OTR, in conducting three FCEs. Each of the three therapists assumed the role of lead therapist for a different client, who underwent his or her FCE while the other two therapists watched and recorded their own scores and ratings independently of the lead therapist. (At each clinic the three therapists met subsequent to the three FCEs to compare their scores and ratings and reconcile, where desired, incongruous scores or ratings. This was done to enhance future Joule training; the analyses reported here pertain only to the initial, independent scores and ratings.) The data for both clinics was combined for the following analyses.

Joule Protocols

The Joule system contains a number of standard protocols in several sections that are administered as appropriate to the needs of clients. These are:

Weighted Tasks

Position Tolerances

Repetitive Tolerances

Upper Extremity


Subjective Client Input

For various reasons, clients sometimes put forth less effort than they could during the FCE. Since it is the purpose of the FCE to determine the functional work capacities of clients, therapists must be able to gauge the degree to which they believe clients have demonstrated their actual capacities during the FCE process. At several points during the Joule FCE, therapists gather information from clients concerning their symptoms, perceived exertion, and perceived safety. One of these points is at the termination of each task. This information is gathered, in part, to evaluate the congruity of the client's reports in light of his or her diagnosis, functional and medical history, etc. in order to evaluate the clientís sincerity and level of effort.

In the Joule system, therapists determine the level of client sincerity of effort in part (see "Participation Index" below) by calculating the percentages of the clientís subjective input falling into several categories. After each work task, the therapist evaluates the quality of client input and assigns to it either a "meaningful," "relevant," or "extraneous" rating.

"Meaningful" ratings are assigned when clients provide information that helps the evaluator determine safe levels of maximum function. Such information is specific and corresponds to the clientís diagnosis, the activity being assessed, and the clientís body mechanics. Meaningful reports are timely, and they directly correspond to objective performance. Meaningful reports are viewed as valid information for determining functional abilities in specific activities and overall tolerances.

"Relevant" ratings are assigned when client reports are of a lower level of importance. The information is generally appropriate to client diagnosis and test performance, but may not be specific in nature. The information may not necessarily be timely; it may be provided by the client during the protocol or at its termination. The reports generally correspond with objective client performance, and are viewed as valid adjunct information to be used subordinately to primary objective criteria.

"Extraneous" client reports provide useless information. They are usually non-specific, and the same report may be offered several times throughout the FCE without reflecting any change in symptoms. These reports may or may not correspond to the clientís diagnosis or the protocol activity. They are often offered several minutes after the work activity has ended, and they have little or no relationship to objective functional performance. They may directly contradict performance or reports in other areas of the FCE. These reports are not considered in determining the clientís functional capacities.

In cases in which no feedback is noted, the rating "no subjective information" is given.

The consistency of the Subjective Client Information ratings was analyzed as follows. In cases in which none of the ratings was the same, a value of zero was recorded (0/3). When two of the three therapistsí ratings were the same, a value of .66 was recorded (2/3); and when all three ratings were the same, a value of 1.0 was recorded (3/3). The total of these consistency values recorded for all of the clients was divided by the number of work tasks rated. The result was converted into a percent, and, in the case of the Subjective Ratings of Client Input, was 84.2%.

 

Participation Index

The "Participation Index" of Joule estimates the clientís sincerity of effort in the FCE and is determined, in part, by a calculation of the percent of "subjective client input ratings" in the several categories described above. Seventy-five percent or more "meaningful" ratings indicates a "consistent" performance, one in which the client was cooperative, one in which his or her subjective input matched the available objective data, and one that indicates that the FCE is a valid representation of the clientís safe work abilities. A "marginally consistent" determination results from an FCE in which from 50% to 74% of the subjective indicators matched the objective findings. The "marginally consistent" determination means that the FCE contained occasional inconsistencies; although the FCE results represent safe work abilities, the clientís performance was self-limiting to a degree and his or her abilities may actually be higher than those demonstrated. An "inconsistent" determination indicated frequent inconsistencies in client performance, and the FCE, in those cases, is not considered to be a valid representation of the client's true abilities. "Inconsistent" determinations are made in cases in which less than 50% of the subjective indicators matched objective findings.

The agreement index for Jouleís Participation Index was calculated as follows. The three possible determinations were treated as an interval scale, from "consistent," to "marginally consistent," to "inconsistent. " One degree of difference was assigned to comparisons in which a therapist had given a "consistent" rating and another therapist had given the "marginally consistent" determination. Two degrees of difference would be assigned if a therapist made a "consistent" determination and another had made an "inconsistent " one. One degree of difference would be assigned if a therapist had made a "marginally consistent" determination and another had made an "inconsistent" one. (See Figure 1.)

 

Figure 1


CMI
C012
M
01
I

0

At both clinics, each therapistís determination per client was compared with those of both of the other therapists. In this approach, there were 4 possible degrees of difference per client per therapist, for a total of 24 possible degrees of difference. A total of 2 degrees of difference were found. The agreement index was 91.7% (agreement index = (n-d/n)x100, where n = number of possible degrees of difference, and d = actual degrees of difference).

In the one case in which there was a difference in the Participation Index, it was between one therapist and the other two at one degree of difference (for a total of 2 degrees of difference) and was due to a calculation error concerning the percent of subjective information ratings on the part of the therapist.

 

Terminating Factors

FCE work tasks are terminated for a variety of reasons, either by the therapist or the client. In some cases, the task has simply been completed. In others, the task is stopped because the therapist notices indications of unsafe behavior or that the client has reached his or her maximum safe limit.

Therapists list the reason for terminating the work task on the "FCE Participation Summary" sheet next to each administered work task. The agreement index for the "Terminating Factors" was calculated in the same manner used to calculate it for the "Subjective Client Ratings" discussed above. If the "Terminating Factors" listed were different among all three therapists, the value of 0 was recorded (0/3); if two of the three therapists agreed, a value of .66 was recorded (2/3); and if all three therapists agreed, a value of 1.0 was recorded (3/3). The total of the recorded values for all clients was divided by the total number of administered work tasks to get the agreement index. In this case, a total of 122.52 recorded values was divided by a total of 131 administered exercises (times 100) to get an agreement index of 93.5%.

 

Last Safe Weight

The "Last Safe Weight" is listed for the lift work tasks only. (See list above.) In Joule, clients are assigned to one of three different weight progression categories. In the first category, clients start the lifts at 2.5 pounds. Lifts started at level 1 progress in 1.25 pound increments until the exercise is stopped. At level 2, lifts start at 5 pounds and progress in 2.5 pound increments. Level 3 clients start their lifts at 10 pounds and progress in 5 pound increments.

The agreement index for the "Last Safe Weight" element of Joule was calculated in a manner similar to the one for the "Participation Index." In this case, differences in the units of the pertinent weight progression for each client were recorded. For example, if one therapist listed "25 pounds" for a client in level 2, and another therapist listed "30 pounds," the difference value between the weights given would be "2" because level 2 clients increase their lifts in increments of 2.5 pounds and 30-25 = 5; and 5/2.5 = 2. The third therapistís listed weight would then be compared to those of the other two, and the units of difference recorded. Note that even if the third therapistís listed weight was the same as one of the others, there would automatically be a total of twice the units of difference recorded for the comparison between the first two therapists.

The total number of recorded units of difference among the three therapists for all six clients was divided by the total number of units of measure administered to all six clients. In this case, there were a total of 34 units of difference and a total of 1122 total units of measure. The agreement index for "Last Safe Weight" was 96.9%. (See formula given for the calculation of the "Participation Index" above.)

Job Frequency Recommendations

"Job Frequency Recommendations," for the purposes of this report, pertain only to the administered work tasks of the "Position Tolerances" and "Repetitive Tolerances." In Joule, the "Job Frequency Recommendations" for the other exercises are determined by a mathematical function based on the "Last Safe Lift," and so there would be no variation possible in those recommendations beyond that already included under the "Last Safe Weight" element.

The Joule system uses modified "Job Frequency Recommendations" based upon the system described in the Revised Handbook For Analyzing Jobs and the Dictionary of Occupational Titles of the U.S. Department of Labor, both published last in 1991. Jouleís system accommodates "Never, " "Rare," "Occasional," "Frequent," "Continuous, " and "No Identified Limitations."

The agreement index for "Job Frequency Recommendations" was calculated in a manner similar to the one used for the "Participation Index, " using a degrees of difference approach. The several frequency ratings possible were placed on an interval scale and the following degrees of difference among the several ratings were assigned.

Figure 2


NVRROFCNL
NVR012345
R
01234
O

0123
F


012
C



01
NL




0


A total of 10 degrees of difference were possible per administered work task. There were a total of 82 administered work tasks within this element. The total number of degrees possible (820), was divided into the total number of recorded degrees of difference, 55. The result was 93.3% agreement. (See the formula given above under "Participation Index".)

 

Summary

Five key elements of Joule were analyzed for interrater reliability. A total of six clients at two clinics underwent FCEs. At each clinic, three therapists participated, each therapist taking the lead role with one client while the other two observed and independently recorded scores and ratings.

The following list shows the Joule elements that were analyzed and the agreement percent index that was calculated for each.



Conclusion

Interrater reliability is crucial to FCEs (see King, Tuckwell, and Barrett, 1998, for example). This study found high levels of interrater reliability in all of the several Joule elements analyzed, as expressed in the various agreement indices used.



Acknowledgements

We wish to thank RSI and Nova Care for their cooperation and support. Particular thanks are due to Kristin Larson, PT and Alison Smetana, OTR/L of RSI; and Lynn Erikson, PT and Sue Anderson, OTR, CHT of Nova Care.



References

Christopherson, B.B. & Ruprecht, M. E. (1999). Joule by Valpar: Introduction and preliminary report of Joule effectiveness and client satisfaction. Tucson: Valpar International Corporation.

King, P.M., Tuckwell, N., & Barrett, T.E. (1998). A critical review of functional capacity evaluations. Physical Therapy, 78, 852-866.

United States Department of Labor, Employment and Training Administration. (1991). Dictionary of occupational titles, (4th ed.) (rev.) Washington, DC: U.S. Government Printing Office.

United States Department of Labor, Employment and Training Administration. (1991). The revised handbook for analyzing jobs. Washington, DC: U.S. Government Printing Office.

 

 

SUMMARY OF FUNCTIONAL ABILITIES

Client Name:

Diagnosis:

Physician:

Date of FCE:

Evaluator:

Social Security Number:

Date of Birth:

 

ABILITY (expressed in % of 8 hour per work day)

ACTIVITY

RARE

1-5%

OCCASIONAL

6-33%

FREQUENT

34-66%

CONTINUOUS

67-100%

NO IDENTIFIED LIMITATIONS

Weighted tasks:

 

 

 

 

 

 

Waist to shin

#

#

#

#

 

Waist to waist

#

#

#

#

 

Waist to eye

#

#

#

#

 

Unilateral carry (R

#

#

#

#

 

Unilateral carry (L)

#

#

#

#

 

Bilateral carry

#

#

#

#

 

Push

#

#

#

#

 

Pull

#

#

#

#

 

Positional Tolerances:

 

 

 

 

 

Kneel

 

 

 

 

 

Crouching

 

 

 

 

 

Mid level reach

 

 

 

 

 

 

Elevated reach

 

 

 

 

 

Sit

 

 

 

 

 

Stand

 

 

 

 

 

 

 

 

 

 

 

Repetitive Tolerances:

 

 

 

 

 

Walking

 

 

 

 

 

Stair Climbing

 

 

 

 

 

Ladder Climbing

 

 

 

 

 

Balance

 

 

 

 

 

 

 

 

Page two

Re:

ACTIVITY

RARE

1-5%

OCCASIONAL

6-33%

FREQUENT

34-66%

CONTINUOUS

67-100%

NO IDENTIFIED LIMITATIONS

Squatting

 

 

 

 

 

Repetitive Foot Motions

 

 

 

 

 

Crawl

 

 

 

 

 

Upper Extremity:

 

 

 

 

 

Grip (R)

#

#

#

#

 

 

Grip (L)

#

#

#

#

 

Coordination (R)

 

 

 

 

 

Coordination (L)

 

 

 

 

 

Work day tolerance:

 

 

 

 

 

The Valpar FCE was administered in 5 hours over 2 days.

Key: No identified limitations = >33% of work day but no specific maximum is determined.

*Indicates a match between physical abilities and job demands.

**Indicates a probable match between physical abilities and job demands.

Job demand information sources:

 

____________________________________ ____________________

Evaluator Signature...........................................................Date

_____________________________________ ____________________

Physician Signature...........................................................Date

cc:

 

 

 

FCE PARTICIPATION SUMMARY

Client Name:

Evaluator:

Physician:

Dates of FCE:

 

Social Security Number:

Date of Birth:

ACTIVITY

TERMINATING FACTOR

SUBJECTIVE CLASSIFICATION

SUBJECTIVE REPORT OF SAFETY

RECOMMENDED TECHNIQUE MODIFICATION

Weighted tasks:

Waist to shin

 

 


 

 

Waist to waist

 

 


 

 

Waist to eye

 

 


 

 

Unilateral carry (R)

 

 


 

 

Unilateral carry (L)

 

 


 

Bilateral carry

 

 


 

Push

 

 


 

Pull

 

 


 

Positional Tolerances:

Kneel

 

 


 

Crouch

 

 


 

 

Mid level reach

 

 


 

Elevated reach

 

 


 

 

Sit

 

 


 

Stand

 

 

 


 

Page two

Re:

ACTIVITY

TERMINATING FACTOR

SUBJECTIVE CLASSIFICATION

SUBJECTIVE REPORT OF SAFETY

RECOMMENDED TECHNIQUE MODIFICATION

Repetitive Tolerances:

Walk

 

 


 

Stair Climbing

 

 


 

 

Ladder Climb

 

 


 

 

Balance

 

 


 

 

Squatting

 

 


 

 

Repetitive Foot Motions

 

 


 

Crawl

 

 


 

 

Upper Extremity:

 

Grip (R)

 

 


 

 

Grip (L)

 

 


 

 

Coordination (R)

 

 


 

 

Coordination (L)

 

 


 

Explanation

Terminating Factor: Reason the test was ended.

Subjective Classifications: The correlation between the client's subjective input and objective findings. Each activity is rated with one of four designations:

1.Meaningful: This information is a valid component of information for determining functional abilities It fully correlates with objective information.

2.Relevant: This information is a valid adjunct to primary objective criteria used in determining functional abilities. It generally correlates with objective information.

3.Extraneous : Do not consider this information as a valid tool in determining functional abilities. No correlation with objective findings.

4.No Subjective Information: Test completed without subjective input.

Subjective Report of Safety: Does the client think the test was safe and reasonable?

Recommended Technique Suggestions for improving safety and body mechanics.

Modification:

Comments:

Recommendations
_____________________________ ______________________
Evaluator Signature Date


Joule Home Page/ Back to Articles/ Home

Copyright © 1999 - Valpar International Corporation; All Rights Reserved.