Measuring the Usefulness and Importance of Outcome Measures
March 2014 Issue
While physicians are tasked by the Centers for Medicare & Medicaid Services (CMS) to assign K-level functional classifications to lower-limb amputee patients, some turn to prosthetists for guidance. Compassion, and even profit margin, can drive prosthetists then to suggest to the physician that he or she assign a lower-limb amputee patient with a higher K-level based on potential rather than reality. If that occurs, the patient has to live with the consequences, be it a device that is too cumbersome or perhaps too heavy. Conversely, if the K-level determination errs on the side of caution and the patient is provided a lower-functioning technology, then his or her performance will be limited, says Brian Kaluf, BSE, CP, managing practitioner of the Ability Prosthetics & Orthotics Greenville, South Carolina, patient care facility and Ability's clinical outcome and research officer. To address the ambiguity surrounding support for K-level assignment, reduce subjectivity, and better serve its patients, Ability, headquartered in Exton, Pennsylvania, instituted a policy whereby all prosthetic patients who visit any of its ten patient care facilities are evaluated using prosthetic outcome measures.
Patient comorbidities, motivation, muscle strength, and sometimes body weight and age can all factor into K-level assignment and inform the prosthetic prescription, Kaluf explains. So what looks like a "cookie cutter approach" to K-levels can be far from that. "For each K-level, there are a few sentences of descriptors and a few factors that Medicare uses to differentiate among patients that have different K-levels, and they end up being overgeneralized and in some ways very specific," Kaluf says. "For instance, 'varied cadence.' What defines varied cadence, and how are you sure that your patient does walk with varied cadence?"
Often when new protocols are instituted, "the final check is to look back and basically learn a little bit," Kaluf says. Thus, last July, the company conducted a retrospective chart review of its prosthetic patients' data to determine if the outcome measures used were indeed beneficial and useful in determining K-levels. On February 28, Kaluf presented the results of the initial six-month retrospective chart review at the American Academy of Orthotists and Prosthetists (the Academy) 40th Annual Meeting & Scientific Symposium, held in Chicago, Illinois.
ASSESSING CLINICAL RELEVANCE
Kaluf joined Ability in September 2011. He earned his prosthetics certificate from the Northwestern University Prosthetics-Orthotics Center (NUPOC), Chicago, Illinois. It was at NUPOC, he says, where he took a research course and "learned everything about psychometric properties and all the different clinically relevant outcome measures." So when he took over operations at the Greenville office in March 2012, he began piloting the use of outcome measures with his prosthetics patients and proposed implementing the measures company-wide to Ability CEO and President Jeff Brandt, CPO. Kaluf received approval and was given the title of clinical outcome and research officer.
"Some subjectivity is appropriate and necessary in the medical field...," Brandt says. "[H]owever, you have to put some science in it...." He has always been interested in measuring outcomes, and the pull to implement them has grown. "[I]n the last few years it became more and more obvious to me that we needed to replace what we were doing in the room with the patient," he says. "If a brand new patient walked into our office, why weren't we going through a protocol with them no different than a doctor goes through a protocol with a patient?"
The protocol that Kaluf instituted is a trio of outcome-based measures, the Amputee Mobility Predictor (AMP), the Prosthesis Evaluation Questionnaire-Mobility Scale (PEQ-MS), and Orthocare Innovations' StepWatch™ Activity Monitor (SAM), or simply StepWatch. Each was chosen, Kaluf says, based on its psychometric properties and ability to help prosthetists best document mobility and support K-level assignment.
The AMP, a series of 20 tasks that measure a patient's potential to ambulate, was the first measure Kaluf says he chose to use because it is one of the most published and accepted measures in prosthetics and was validated by gait training expert Robert S. Gailey, PhD, PT, and his colleagues. (Gailey, R. S., et. al, "The Amputee Mobility Predictor: An Instrument to Assess Determinants of the Lower-Limb Amputee's Ability to Ambulate," Archives of Physical Medicine & Rehabilitation, Vol. 83, May 2002.) "It's been around for a while, and everyone really understood that acronym pretty well and what the score in the Amputee Mobility Predictor represents...[and it is] a tool that could prove our [K-level] assignment immediately," Kaluf says. The AMP is particularly helpful, he stresses, because it is the only outcome measure that can assess ambulatory potential of lower-limb amputee patients with a prosthesis (AMPPRO) and without a prosthesis (AMPnoPRO). Results can be quantified.
Figure 1: Amputee Mobility Predictor with Prosthesis (AMPPRO) mean scores +/- 1 standard deviation for the different K-levels. The black lines indicate the standard deviations; they show that there is overlap in the groups. For example, a patient may have an AMPPRO score that is higher or lower than the mean for his or her assigned K-level.
The PEQ-MS, on the other hand, is a patient self-report measure. It consists of 12 questions and was designed to assess a patient's locomotor ability while using a prosthesis. Ability's decision to use the PEQ-MS was based on similar reasoning as its decision to use the AMP: Its validity has been proven, and it is reliable, widely recognized, and well published. "[T]here was a lot of time spent in developing [the PEQ], and it is used frequently in research to validate other measures," Kaluf says. "Sometimes that patient's perspective of their ambulatory capability is just as important of a measure as the more timeconsuming gait analysis. It basically gives you insight into limitations that they have in the community."
The final outcome measure, admittedly not used on all prosthetics patients, Kaluf says, is the StepWatch, which, with Orthocare Innovation's Galileo software, can record, document and assess the subject's gait. "It's performance based, it's proven... and it also shows utilization," Kaluf explains. "The Galileo is a higher level of assessment that we use [only] on patients that need a higher level of scrutiny to help guide their treatment pathway or if we expect that their payer source is going to have higher scrutinization of a certain technology."
APPLYING THE MEASURES
"Each measure has instructions that tell you when and how they are appropriate to use," Kaluf says, explaining that Ability clinicians administer the standardized outcome measures during the initial evaluation for a new prosthesis, socket replacement, or new prosthetic foot component. At periodic follow-up intervals, he adds, the clinician may administer a measure to see if the patient has progressed or regressed in his or her mobility. "So a new amputee would only have an AMPnoPRO score. But a mature amputee receiving an update in prosthetic foot technology or a new socket would have AMPPRO, PEQ-MS, and perhaps StepWatch data."
Figure 2: Prosthesis Evaluation Questionnaire-Mobility Section (PEQ-MS) mean scores +/- 1 standard deviation for the different K-levels. The black lines indicate the standard deviations; they show that there is overlap in the groups. For example, a patient may have a PEQ-MS score that is higher or lower than the mean for his or her assigned K-level.
By applying multiple outcome measures and reviewing results against published studies, in addition to subjective information in the patient's chart, the prosthetist can make a better-informed K-level assessment, Kaluf explains.
He provides an example of administering the AMP and PEQ-MS to a patient whose test scores show that she is at the higher end of the K2 spectrum. "Then you can go back to all the other subjective and objective information from that patient's chart, and you have a better idea where the patient currently is. If [her scores are on the higher end] of the K2 range compared to published data, and you see that she is improving in her condition, and she has the strength to hop with the walker and is actively participating in therapy and rehab, that gives you a clear assessment that this patient will improve." With improvement, he says, comes increased strength and stability, "which allow the prosthetist to confidently recommend a potential K3 classification for the patient and cite patient-specific outcome data to support the assignment."
THE RESULTS ARE IN
The decision to conduct the retrospective chart review was essentially a quality assessment of the new protocols-to determine if they are useful and important outcome measures to use with the lower-limb amputee population, Kaluf says. The results, which have been accepted for publication in the April issue of the Journal of Prosthetics and Orthotics, provide a real-world clinical setting for these measures. "There is nothing worse than sitting on a bunch of data that could help change your industry and you don't get it out there," Brandt says.
To begin the chart review, Kaluf separated the data of 57 patients into two groups, those with a transfemoral amputation and those with a transtibial amputation. He applied a statistical analysis and found that the AMPPRO and PEQ-MS scores were not statistically different among these two populations. He then separated the patients into groups based on their clinician-assigned K-level. He compared the mean PEQ-MS scores of each K-level group using two different statistical analyses, which indicated that the K3 and K4 groups scored significantly higher on the PEQ-MS than did the K2 group. "My data shows that K2 patients have lower perceived mobility than K3 and K4 patients," Kaluf explains, adding that while it may be obvious, it must nevertheless be measured, documented, and thoroughly understood to manage prosthetic care. "This tells us that the PEQMS can differentiate K2 from K3/K4 patients."
By applying similar statistical analyses to the AMP scores, Kaluf was again able to determine that the K2-level group differed significantly from K3- and K4-level groups, he says. He also compared the AMP scores from his retrospective chart review against the AMP scores recorded in Gailey's clinical trial as additional support. While the average scores of Ability's patients were lower at each K-level assignment compared to Gailey's study cohort, Kaluf attributes the difference to Gailey's study being conducted in a controlled environment, while Ability's study was a review of raw clinical chart data. "Anyone who is a successful prosthetist will attest that the clinic environment is anything but controlled, despite our best efforts," Kaluf says.
So what do these results mean? "My one sentence conclusion is that both the PEQ and the AMP are able to differentiate K3 and K4 patients from K2 patients," Kaluf says. "Ability Prosthetics & Orthotics has found that use of standardized outcome measures as a routine part of our evaluation of patient mobility has improved our skill in assigning patient K-level, and this was supported in the findings of the retrospective chart review."
APPLYING THE RESULTS
Treating the patient according to the appropriate K-level is one of the most important decisions a prosthetist must make with a new patient-and using the wrong functional level, and thereby providing a patient with the wrong prosthesis, can hinder potential mobility. Coupled with the plethora of Recovery Audit Contractor (RAC) audits scrutinizing the provision of K3-level prostheses, and third-party payers following suit, practitioners have all the more reason to apply outcome measures when consulted on functional levels. Successfully navigating the audits, or perhaps bypassing them altogether, comes down to documentation.
The value of outcomes documentation is further increased because it is objective, quantifiable data that can be provided to referral sources to show why a patient might, for example, need an Ottobock C-Leg, and thus why the physician should enter the prosthetist's chart notes as the official record. The documentation could even be provided to the physical therapist to show how the patient is progressing or where there are gaps in the patient's progress, Brandt says.
"For me, it was like I was fitting a $20,000 leg but our [documentation] was worth $1,000," says Brandt of his early years in the profession. "So if we are going to provide a $20,000 leg, then let's provide $20,000 documentation. It is just about producing quality."
"Without quantitative data from standardized outcome measures to support K-level decisions, patient access to prosthetic technology and their potential mobility relies heavily on the perceptions, preconceived notions, and varied experience of the individual clinicians," Kaluf says. "We are absolutely trying to continue to lead the field and provide an example of the level of care and the level of understanding of patients' mobility that can be achieved."
Laura Fonda Hochnadel can be reached at