An Appraisal Standard for a quick and reliable comparison between risk stratification methods for local healthcare systems

Introduction: European countries need to address the issues related to ageing populations and the consequent delivery of health care to multi-morbid and or frail patients. A potential solution is represented by the identification of those patients at greatest risk for costly and preventable service utilization by means of risk stratification tools. Currently, many stratification and predictive modelling software tools exist, with various degrees of (claimed) accuracy, internal/external validation, requirements on the input data, and target population. Because of this scenario, an Appraisal Standard (AS) model which evaluates different stratification techniques has been designed. The AS allows policy makers and health care managers to evaluate different stratification methods and to identify the most suitable for the health care environment under investigation. The work herewith described has been performed within the European Project ASSEHS (http://assehs.eu/). Methods – Literature review and Survey: First, a systematic literature review was conducted to extract data from validations of stratification tools in different populations that have been published in publically available reports or in peer-reviewed journal article resulting in a total of 216 publications or reports. A hand-selection of 26 publications was used to start designing the first version of the AS model. A second approach of data collection was considered to gather that information on either unpublished tools or aspects usually not included in scientific manuscripts (e.g. implementation 15th International Conference on Integrated Care, Edinburgh, UK, March 25-27, 2015 1 International Journal of Integrated Care – Volume 15, 27 May – URN:NBN:NL:UI:10-1-117019 – http://www.ijic.org/ and ease of use of the tool). A survey was developed to be sent out to appropriate professionals associated with one of the health care regions within and outside ASSEHS’ consortium. The surveys are meant to gather hands-on experiences that we could not retrieve from literature searches. The Appraisal Standard: The literature review and survey led to both the design of the data model underneath the AS and the collection of the information stored in the AS’ database. More specifically, the AS which comprises two main parts: an evaluation of the intrinsic properties of the stratification tools themselves and an evaluation of the study conditions that were used to develop and/or validate these tools. As a consequence, the core of the AS contains a table including a variety of properties of a stratification tool (e.g. model name, main technique, outcome type, etc.), and a table describing the different studies that evaluate those tools (e.g. authors, year, study type, measured risk, population, etc.). Multiple studies may have evaluated a particular stratification tool in different populations. Furthermore, one study may have compared a variety of stratification tools under the same conditions. There are also a wide range of evaluation metrics that can be used to determine the performance of stratification tools, such as area under the operator characteristic curve, mean squared prediction error etc. The Appraisal Standard therefore covers a wide range of evaluation metrics generally used in the scientific literature. These evaluation metrics were included in a table, using evaluation metric values as the connection between the tables describing stratification tool and the table describing study conditions. Finally, stratification tools generally make use of the same or similar (categories of) predictor variables (e.g. socio-demographic data, diagnostic data, pharmacy data, etc.). To account for this, the AS includes a table with such variables. The stratification model table can refer to this table. Discussion and conclusion: The work presented here is aligned with the priorities identified within several European health systems towards the implementation of proactive, anticipatory and integrated care. This approach is an answer to the strong urge of health care expenditures’ mitigation, enhancement of quality and continuity of care and health care sustainability. The AS will allow policy makers and health care managers to perform a critical comparison of different risk stratification techniques guiding them towards the selection of the best-in class stratification technique for their health care environment. The aim of this study is twofold: on one hand to support and guide those decision makers already aware of the efficacy of risk stratification techniques; on the other hand, to disseminate the knowledge on risk stratification tools via a comprehensive and unique source of information to reach those professionals and or organizations not yet informed about the deployment of risk stratification techniques. Limitations and future steps: As discussed in the section “Methods”, some health care organization developed their own stratification model but neither technical reports nor peerreviewed scientific articles have been published to describe their tool. For such scenario, we developed a survey to gather the information needed in order to include unpublished current or future tool in the database.

Because of this scenario, an Appraisal Standard (AS) model which evaluates different stratification techniques has been designed. The AS allows policy makers and health care managers to evaluate different stratification methods and to identify the most suitable for the health care environment under investigation. The work herewith described has been performed within the European Project ASSEHS (http://assehs.eu/).

Methods -Literature review and Survey:
First, a systematic literature review was conducted to extract data from validations of stratification tools in different populations that have been published in publically available reports or in peer-reviewed journal article resulting in a total of 216 publications or reports. A hand-selection of 26 publications was used to start designing the first version of the AS model.
A second approach of data collection was considered to gather that information on either unpublished tools or aspects usually not included in scientific manuscripts (e.g. implementation and ease of use of the tool). A survey was developed to be sent out to appropriate professionals associated with one of the health care regions within and outside ASSEHS' consortium. The surveys are meant to gather hands-on experiences that we could not retrieve from literature searches.

The Appraisal Standard:
The literature review and survey led to both the design of the data model underneath the AS and the collection of the information stored in the AS' database. More specifically, the AS which comprises two main parts: an evaluation of the intrinsic properties of the stratification tools themselves and an evaluation of the study conditions that were used to develop and/or validate these tools. As a consequence, the core of the AS contains a table including a variety of properties of a stratification tool (e.g. model name, main technique, outcome type, etc.), and a table describing the different studies that evaluate those tools (e.g. authors, year, study type, measured risk, population, etc.).
Multiple studies may have evaluated a particular stratification tool in different populations. Furthermore, one study may have compared a variety of stratification tools under the same conditions. There are also a wide range of evaluation metrics that can be used to determine the performance of stratification tools, such as area under the operator characteristic curve, mean squared prediction error etc. The Appraisal Standard therefore covers a wide range of evaluation metrics generally used in the scientific literature. These evaluation metrics were included in a table, using evaluation metric values as the connection between the tables describing stratification tool and the table describing study conditions. Finally, stratification tools generally make use of the same or similar (categories of) predictor variables (e.g. socio-demographic data, diagnostic data, pharmacy data, etc.). To account for this, the AS includes a table with such variables. The stratification model table can refer to this table.

Discussion and conclusion:
The work presented here is aligned with the priorities identified within several European health systems towards the implementation of proactive, anticipatory and integrated care. This approach is an answer to the strong urge of health care expenditures' mitigation, enhancement of quality and continuity of care and health care sustainability. The AS will allow policy makers and health care managers to perform a critical comparison of different risk stratification techniques guiding them towards the selection of the best-in class stratification technique for their health care environment.
The aim of this study is twofold: on one hand to support and guide those decision makers already aware of the efficacy of risk stratification techniques; on the other hand, to disseminate the knowledge on risk stratification tools via a comprehensive and unique source of information to reach those professionals and or organizations not yet informed about the deployment of risk stratification techniques.
Limitations and future steps: As discussed in the section "Methods", some health care organization developed their own stratification model but neither technical reports nor peerreviewed scientific articles have been published to describe their tool. For such scenario, we developed a survey to gather the information needed in order to include unpublished current or future tool in the database.