Clinical prediction rules

INTRODUCTION — A clinical prediction rule (CPR), if well-designed and validated, provides patient-specific evidence, quantifying the roles of history, physical examination, and laboratory results in diagnosis, prognosis, and likely response to treatment [1]. When available at the point of care, CPRs can allow clinicians to blend clinical acumen and evidence in real time. Clinicians may find CPRs useful when decision-making is complex, risks are high, or there are opportunities to reduce costs without compromising patient care [1]. Ideally, a CPR used in a clinical setting has undergone rigorous development and testing.

There are relatively few highly validated CPRs in each specialty. Therefore, for each clinician there may only be one or two relevant CPRs for day-to-day clinical practice. It is important to understand the validity and limitations of those CPRs that each clinician may use. This topic reviews how to evaluate a CPR.

Other aspects of evidence-based medicine are discussed elsewhere.

●(See "Glossary of common biostatistical and epidemiological terms".)

●(See "Systematic review and meta-analysis".)

●(See "Evidence-based medicine".)

●(See "Real-world evidence in health care".)

TERMINOLOGY — Various terms such as risk score, risk calculator, algorithm, and prognostic model are often used interchangeably with CPRs. Throughout this topic, we will use the term clinical prediction rule, which is based on real-time patient data points such as medical history, physical examination, and laboratory data. CPRs weigh patient data and generate a composite score to stratify patients' risk of disease onset, disease progression, or outcomes events. The use of CPRs generates a calculated number and/or score that reflects the severity of a risk due to various predefined factors. These rules are patient-specific and provide specific recommendations for rapid frontline decision making.

EVALUATING CLINICAL PREDICTION RULES FOR CLINICAL USE

Approach — There are three sequential steps in the process of creating and validating a CPR:

●Derivation – The first step in the process is the creation, or derivation, of the rule.

●Validation – This step involves assessing how accurate the rule is when used in clinical practice.

●Impact analysis – The final step is the impact analysis, which assesses the impact the rule has when applied to a population compared with when it is not applied.

Each of these steps may be published separately, and understanding the validity and quality of each step is important. In each of these steps there are keys to performing a high-quality study, as outlined in the sections below.

Derivation — This is the step in which a CPR is created. CPRs are initially derived by first creating a list of pertinent components of a patient's past history, current symptoms, physical examination findings, and readily available laboratory results which are thought to be correlated with the outcomes of interest (ie, broken ankle, bacterial infection, deep venous thrombosis, etc). The list of predictors to be evaluated is typically generated by subject matter experts. The list of predictors is then either retrospectively evaluated using a database or prospectively evaluated correlating those predictors to the outcome of interest. Those predictors which are found to be correlated with the outcome are then used to create the CPR.

There are important standards that need to be considered for the derivation of a CPR:

●Were all appropriate predictors included in the evaluation?

●Were those predictors present in a significant amount of the study population?

●Did the study population resemble the population in which the rule is likely to be applied?

●Were the outcome events and predictors clearly defined?

●Were those assessing the presence of the outcome event blinded to presence of predictors?

●Does the rule make clinical sense and is easy to use and calculate in practice.

Validation — Validation refers to assessing how accurate a CPR is when applied in clinical practice. Validation of CPR can be performed either retrospectively by applying the rule to a previously collected large data set or, preferably, prospectively to a new set of patients. The ultimate validation is performed when a rule is applied prospectively multiple times. In each case the rule is evaluated for its accuracy such as positive and negative predictive value.

Impact analysis — The final step in the evaluation process of a CPR is to perform an impact analysis, which assesses the impact the rule has when applied to a population compared with when it is not applied. This final step is rarely performed but is critical to assess the true benefits of implementing a CPR. Answering important questions such as does applying the rule reduce unnecessary testing or treatment and other patient important outcomes can be best assessed such as quality of life, morbidity, and resource utilization. The impact analysis is not for accuracy, but rather for assessing change in clinical behavior.

Examples

Ottawa ankle rules — The Ottawa ankle rule is a CPR that identifies patients with ankle injuries who are at high risk for having a fracture, thereby assisting clinicians in being more selective in their use of radiography. (See "Ankle sprain in adults: Evaluation and diagnosis", section on 'Ottawa ankle rules'.)

●Derivation – The Ottawa ankle rule is a well derived CPR [2]. The rule met all quality standards for a derivation; specifically, all important predictors were evaluated, according to subject matter experts; the predictors and outcomes were well defined; and the assessment of the outcome event of a malleolar fracture was interpreted by qualified radiologists who were blinded to the predictors. An adequate sample size, as determined by the anticipated frequency of events, of over 750 patients was collected.

●Validation – In the prospective validation of the Ottawa ankle rule, patients were chosen from a sample of adults presenting to the emergency department with acute ankle injury [3]. Patients were assessed for the predictors found in the derivation study and assessed for fracture by radiograph. Those assessing the radiograph were blinded to the results of the CPR.

In the validation study, the rules had a sensitivity of 1 for malleolar zone fractures, and 1 for midfoot zone fractures. The possible reduction in radiography was estimated as 34 percent for the ankle series and 30 percent for the foot series. The probability of fracture for those with a "negative" result by the Ottawa ankle rule was estimated as 0 percent in the ankle series and 0 percent in the foot series.

●Impact analysis – In the Ottawa ankle rule an impact study was performed, with confirmation of a change in clinical behavior. As such, we consider this to be a CPR with high-quality supporting evidence. Note that the goal of impact analysis is to assess whether a validated CPR changes clinical behavior, not to assess accuracy (which is done in the validation step.)

A multicenter randomized controlled trial was conducted, followed by observational studies of radiologic practices [4]. The study took place in the emergency departments of five hospitals including adults with acute ankle or midfoot injuries, and demonstrated a reduction in ankle radiography in the intervention group compared with the control group (79 percent versus 99 percent).

Well's criteria for pulmonary embolism — The Wells' clinical model for estimating the probability of pulmonary embolism was derived in 1998 [5] and is still frequently used. (See "Clinical presentation, evaluation, and diagnosis of the nonpregnant adult with suspected acute pulmonary embolism", section on 'Determining the pretest probability of pulmonary embolism'.)

●Derivation – The rule included all important predictors (as determined by subject matter experts), the predictors and outcome event were well defined, and confirmatory scans for pulmonary embolism were completed by clinicians with no knowledge of predictors [5]. An adequate sample of 1239 inpatients and outpatients were included in the study.

●Validation – Prospective validation was performed on all eligible patients referred for pulmonary embolism imaging following the provider's intake of history and physical examination, chest radiograph, and electrocardiogram [6]. Patients were evaluated for each component of the Well's criteria and researchers were appropriately blinded. All patients were evaluated with either a ventilation/perfusion lung scan or a helical CT angiogram of the chest. If the patient was not diagnosed upon initial presentation with a pulmonary embolism, then patients were followed-up with a three-month telephone interview.

Among 930 patients presenting to four emergency departments with suspected pulmonary embolism, the pretest probability of pulmonary embolism using Well's criteria was low, moderate, and high in 527 (57 percent), 339 (36 percent), and 64 patients (6.8 percent), respectively [6]. The rates of pulmonary embolism were 1.3 percent for those with low pretest probability; 16 percent for those with moderate probability; and 38 percent for those with high probability. The negative predictive value exceeded 99 percent.

●Impact analysis – Randomized trials have not been conducted to confirm the clinical benefit of using Well's criteria in patients with suspected pulmonary embolism, limiting the quality of the supporting evidence. Nevertheless, given the robust derivation and validation steps, we continue to use it to provide an estimate of the pretest probability of pulmonary embolism.

A meta-analysis of observational studies has evaluated the impact of Well's criteria and found a modest clinical benefit [7]. In this meta-analysis, data from four studies assessing the Well's criteria were pooled. The frequency of pulmonary embolism by CT angiography was 12 percent in the Well's criteria group and 9 percent in the control group, for a 3 percent increase in diagnostic yield when Well's criteria were applied (95% CI 1-5 percent; relative risk 1.3, 95% CI 1.1-1.6). Of two studies that provided sufficient information for analysis, there was no difference in the rate of missed pulmonary embolism between the intervention and control groups.

IMPLEMENTATION OF CPRS — Even well validated CPRs can only be useful if they are adopted and used in clinical practice. To enhance adoption, it would be helpful to integrate them seamlessly into the electronic medical records (EMR). There are three key steps to integration of CPRs into the EMR.

●Usability – Usability testing serves as a "near live" clinical laboratory where end users are observed interacting with the CPR integrated within the EMRs and/or scripted simulations of patient encounters where the CPR is utilized. The observations are then analyzed with thematic analysis procedures. The focus of usability testing is to assess the efficiency, usefulness, information content, user interface, and workflow components of the CPR. It is essential to analyze each environment to understand unique cultural aspects of interactions among the patient, provider, and EMR [8].

●Adoption – EMR reporting systems provide details about the frequency of CPR alerts and adoption rates for each step of a CPR which include the alert, CPR completion, automatic order set, and documentation. Adoption assessment includes such elements as provider compliance with the recommendations and corresponding order set following risk assessment.

●Postsurveillance monitoring – Close postsurveillance monitoring of CPRs may help improve provider adoption. Adaptive modifications based on user feedback may increase targeted CPRs with lower alerts, reducing alert fatigue and increasing provider adoption. Iterative improvements and a postsurveillance monitoring dashboard can significantly improve adoption rates of a CPR [9].

SPECIAL CONSIDERATION — More and more CPRs are being integrated into electronic medical records (EMRs) through clinical decision support mechanisms. Understanding the validity of integrated models and where and how to apply them is important as they become integrated. In addition, CPRs are often updated or revalidated and therefore need to be updated in the EMR.

Evaluation of CPRs embedded in the EMR can pose a challenge, particularly given that only the outputs are given to clinicians. Some CPRs are derived and validated retrospectively using forms of machine learning (ML). In addition, these models may adjust their predictive accuracy as more data is collected over time. There is no clear consensus on how validity is ultimately assessed in these ML derived and continuously learning models. Concerns of a "black box" phenomenon exist, where clinicians have no way of assessing the actual rule itself or how it is validated and continually updated. Performing prospective impact analysis of applying the ML derived and integrated CPRs compared with not applying the ML derived rule would be the ideal method to assess the true accuracy and impact, but as mentioned earlier, these studies are rarely performed. (See 'Impact analysis' above.)

FUTURE DIRECTIONS — Standard CPRs will continue to be assessed in the same fashion using techniques for derivation, validation, and impact analysis. Machine learning will play a larger role and could enable more rapid derivation and validation, but the process of validation and standards are not yet determined.

SUMMARY

●Introduction – There are relatively few highly validated clinical prediction rules (CPRs) in each specialty. Therefore, for each clinician there may only be one or two relevant CPRs for day-to-day clinical practice. It is important to understand the validity and limitations of those CPRs that each clinician may use routinely. (See 'Introduction' above.)

●Evaluation of clinical prediction rules – CPRs may be published in varying stages of their evolution: derivation, validation, or impact analysis.

•Derivation – This is the step in which a CPR is created. CPRs are initially derived by first creating a list of pertinent components of a patient's past history, current symptoms, physical examination findings, and readily available laboratory results which are thought to be correlated with the outcomes of interest. (See 'Derivation' above.)

•Validation – Validation refers to assessing how accurate a CPR is when applied in clinical practice. Validation of CPR can be performed by either retrospectively applying the rule to a previously collected large data set or, preferably, prospectively to a new set of patients. (See 'Validation' above.)

•Impact analysis – The final step in the evaluation process of a CPR is to perform an impact analysis, which assesses the impact the rule has when applied to a population compared with when it is not applied. This final step is rarely performed but is critical to assess the true benefits of implementing a CPR. (See 'Impact analysis' above.)

It is important to assess what stage a CPR is prior to applying it into clinical practice. (See 'Evaluating clinical prediction rules for clinical use' above.)

●Implementation of clinical prediction rules – Even well validated CPRs can only improve outcomes if they are adopted and used in clinical practice. To enhance adoption, it would be helpful to integrate them seamlessly into the electronic medical records. (See 'Implementation of CPRs' above.)

●Future directions – Machine learning will play a larger role in refining CPRs, and could enable more rapid derivation and validation, but the process of validation and standards when machine learning is involved are not yet determined. (See 'Future directions' above.)