Start Submission Become a Reviewer

Reading: Methodological Quality of Economic Evaluations in Integrated Care: Evidence from a Systemati...


A- A+
Alt. Display

Research & theory

Methodological Quality of Economic Evaluations in Integrated Care: Evidence from a Systematic Review


Mudathira Kadu ,

International Foundation for Integrated Care, Oxford, UK; Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, CA
X close

Nieves Ehrenberg,

International Foundation for Integrated Care, Oxford, GB
X close

Viktoria Stein,

International Foundation for Integrated Care, Oxford, GB
X close

Apostolos Tsiachristas

Health Economics Research Centre, Nuffield Department of Population Health, University of Oxford, Oxford, GB
X close


Introduction: The aim of this review is to systematically assess the methodological quality of economic evaluations in integrated care and to identify challenges with conducting such studies.

Theory and methods: Searches of grey-literature and scientific papers were performed, from January 2000 to December 2018. A checklist was developed to assess the quality of economic evaluations. Authors’ statements of challenges encountered during their evaluations were qualitatively coded.

Results: Forty-four articles were eligible for inclusion. The review found that study design, measurement of cost and outcomes, statistical analysis and presentation of data were the areas with most quality variation. Authors identified challenges mostly related to time horizon of the evaluation, inadequate or lack of comparator group, contamination bias, and a post-hoc evaluation culture.

Discussion: Our review found significant differences in quality, with some studies showing poor methodological rigor; challenging conclusions on the cost-effectiveness of integrated care.

Conclusion: It is essential for evaluators to use best-practice standards when planning and conducting economic evaluations, in order to build a reliable evidence base for decision-making in integrated care.

How to Cite: Kadu M, Ehrenberg N, Stein V, Tsiachristas A. Methodological Quality of Economic Evaluations in Integrated Care: Evidence from a Systematic Review. International Journal of Integrated Care. 2019;19(3):17. DOI:
  Published on 09 Sep 2019
 Accepted on 20 Aug 2019            Submitted on 25 Feb 2019


Health care systems around the world are seeking strategies that improve efficiency in the organization and delivery of care to meet increasing care needs and rising healthcare costs [1]. In the United States, failure to coordinate care; particularly for populations with chronic illness, cost the healthcare system about $35 billion to $45 billion in 2011 [2]. Integrated care is one of the most prominent strategies which have been proposed as a means to improve population health, organize complex care and control healthcare cost growth [1, 3]. The recent paradigm shift in care delivery, from acute and segmented care, to preventive and coordinated care, is in alignment with the WHO Framework on Integrated People-centred Health Services. This framework calls for more integrated approaches to how services are funded, managed and delivered [4].

The process of integrating care is not straightforward and dependent on several factors such as availability of resources, health and social care systems, and commitment and support of stakeholders [1, 4]. As a result, different types of integrated care models have been designed and implemented around the world. These differ by the type of population served, care sectors and services integrated, care providers involved, and health care financing and reimbursement structures set in place [1]. However, despite the increasing implementation of integrated care models, the scientific evidence on its cost-effectiveness remains ambivalent [5].

Economic evaluations examining whether integrated care interventions can achieve value for money, are becoming increasingly common alongside larger evaluation studies [6]. The economic evidence that has been summarized in systematic reviews and meta-analyses is generally inconclusive [8, 9, 10, 11, 12, 13, 14]. A review by De Bruin et al. reported that 62% of disease management programmes were cost saving [13]. A recent meta-analysis by Desmedt et al. found that 50% of integrated care interventions for multiple sclerosis, 67% for schizophrenia and 89% for diabetes were cost-effective [9]. Ouwens et al found that 57% of the studies on integrated care interventions were cost-saving [8]. There are many possible explanations for the inconclusiveness. This may include the widely varying definitions and components of integrated care programmes implemented and measures of impact chosen in the evaluation [5, 7, 8, 15]. Yet, the most commonly suggested reasons are related to the methodological quality of the evaluations [5, 15]. In their review of systematic reviews on this topic, Nolte and Pitchforth concluded that “the quality of existing economic evaluations is the main impediment to arriving at a robust evidence base that is suitable to inform decision making in integrated care” [5, p. 37]. Despite best practice recommendations [16, 17], very few have conducted quality assessment of studies included in systematic reviews examining the impact of integrated care [9, 12, 13, 18]. Little is known about current practices in conducting economic evaluation of integrated care interventions, or the methodological gaps related to the overall study designs, measurements of impact, analysis and presentation of results.

Results of economic evaluations are increasingly being used to inform healthcare decision-making, which further underscores the need to examine whether these evaluations meet methodological standards [19]. Policy makers concerned with the transferability of integrated care models across healthcare settings may be interested in knowing whether the conclusions expressed in studies are acceptable and trustworthy [20, 21]. This is especially the case since allocating public funds to healthcare services informed by misleading or biased economic analysis, may pose significant ethical issues and policy implications [19, 22]. Employing reliable methods in economic evaluations may also be of importance to clinicians and health practitioners considering integrated care in their health service planning and service delivery [20]. Finally, a systematic assessment of the quality of evidence generated by economic evaluation is important in contextualizing the substantial variation in findings of integrated care interventions, as well as in identifying opportunities for improvements [5]. Therefore, the aim of this review is to systematically assess the methodological quality of economic evaluations in integrated care and to identify challenges with conducting such studies.


Search strategy

Relevant search terms related to the broad concepts of “integrated care” and “economic evaluation” were identified by looking at frequently used terms in previous systematic reviews [5, 7, 9, 10, 12, 14] and seminal literature [17, 19, 20, 21, 23, 24] on the respective topics. Our search was also informed by terms used in the Integrated Care Search tool developed by the International Foundation for Integrated Care [25]. See Appendix 1 for search terms used in the review.

Searches of grey-literature and scientific papers were performed, from January 2000 to December 2018, in the following databases: Medline/PubMed, EMBASE, CINAHL, Web of Science, Scopus, the World Health Organization Library and Information Networks for Knowledge database (WHOLIS), Database of Abstracts of Reviews of Effects (DARE) and the NHS Economic Evaluation Database (NHS EED), and the OECD Library. In addition, hand searching of reference lists in key publications (including systematic reviews on integrated care, dissertations, conference proceedings, opinion pieces, editorials and conference abstracts) was used to identify any relevant missed articles.

Selection process and eligibility criteria

Citations were downloaded and screened in Mendeley, an online citation manager tool. All article abstracts, and titles were read independently by two reviewers based on the inclusion criteria detailed below. If the reviewers could not determine whether to exclude an article based on its abstract and title, then it was retrieved for full text reading until agreement was reached.

Inclusion criteria

  1. Articles published in English.
  2. Articles that described the implementation, execution or evaluation of interventions or programmes based on Kodner and Spreeuweberg’s definition of integrated care: “funding, administrative, organisational, service delivery and clinical interventions designed to create connectivity, alignment and collaboration within and/or between the cure and care sectors” [26, p. 3].
  3. Empirical economic evaluations as defined by Drummond et al.: “the comparative analysis, measurement, valuing and identification of alternative courses of actions in terms of their cost and consequences” [20, p. 3].

Data abstraction and analysis

Two data abstraction templates were adapted from Boland et al. [12], and aligned with elements from the PICO framework for research [27]. The first template was used to extract information about the studies including: the study objectives, the settings, description of the intervention, target population and size. The table was also used to abstract information on the evaluation, including study design, the type of economic analysis, the evaluation perspective, the length of the observation period and the measures used.

The second template was developed to assess the quality of economic evaluations. This was in the form of a checklist appraising the economic evaluation methodology and the risk of bias in the study design of the interventions. It provided a binary scoring criterion (yes/no) assessing strengths and weakness of the studies on 30 items. The checklist was a combination of: 1) the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) [17], and 2) the Health Technology Assessment of Disease Management Programmes (HTA-DM) [28]. The CHEERS guideline was developed by an expert panel to optimize the reporting of health economic evaluations [17], while the HTA-DM has been previously used to assess the overall quality of evaluations [28]. Two reviewers (MK, NE) abstracted the information from both templates, and then independently cross-verified each other’s information for reporting consistency and data reliability.

Our approach to identifying challenges and limitations with their overall evaluations was modeled after the thematic analysis approach used in a systematic review by Thomas and Harden [29]. This involved three stages: 1) coding text, 2) developing descriptive themes from lifted texts and 3) generating analytical themes. Firstly, in the coding phase, when articles described barriers, limitations or challenges that emerged from the economic evaluation of integrated care, they were regarded as “attributive statements” and systematically recorded. These statements were often found in the discussion and results section of the articles. In the second stage, descriptive themes that emerged from the lifted texts were identified. Finally, three reviewers (MK, NE and VS) used an inductive approach to identify the dominant themes that emerged from clustering statements of challenges [30].


The initial literature search and screening based on the inclusion criteria yielded 2,263 abstracts. Most of these articles were rejected as they were not based on interventions that fulfilled the definition of integrated care and did not report the results of economic evaluations. After full text reading of 69 articles, 44 articles were eligible for inclusion. Figure 1 shows our PRISMA flowchart at various stages of the selection process.

Figure 1 

Flowchart of study inclusion at various stages of the selection process.

The most common types of economic evaluations were cost-utility analysis [31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49], followed by cost-consequence [50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63], cost-effectiveness [64, 65, 66, 67] and cost-comparison analysis [68, 69, 70, 71, 72]. For more information on the different types of economic evaluations, please see Drummond et al. [19]. Studies conducted economic evaluations in various regions, which were based in North America (n = 16), United Kingdom (n = 2), Europe (n = 20), Australia (n = 1), Africa (n = 3) and Asia (n = 2). The sample size of the studies varied from 36 individuals in the intervention group, to large-scale population-level evaluations of integrated care initiatives. The interventions targeted various groups including older adults with chronic illness or frailty, children, individuals with mental illness and/or substance abuse, individuals treated for multiple occurring chronic conditions or multimorbidity and high users of the healthcare system. Many interventions were delivered across sectors, more commonly integrating acute, primary and secondary care. Others were single-sited and focused on evaluating clinical integration at the multidisciplinary team level. Please refer to Table 1 for description of the studies, and patient and intervention characteristics.

Table 1

Description of study, patient and intervention characteristics.

Study characteristics Patient characteristics

Reference Country Study design Type of economic evaluation Pers-pective Inter-vention Size Control Size Observation period Target population Intervention description Setting Study objective(s) Measures

Zulman et al. (2017) USA RCT Cost-consequence Health care payer 150 433 17 months High healthcare users (top 5%) The ImPACT multidisciplinary team addressed health care needs and quality of life through comprehensive patient assessments, intensive case management, care coordination, and social and recreational services Primary care medical home To evaluate the impact of augmenting the Veterans Affairs’ medical home and multidisciplinary team with an intensive management program Outcomes: 1) patient satisfaction, 2) patient activation measures; Cost: inpatient and outpatient services
Weisner et al. (2001) USA RCT Cost-effectiveness Health care payer 285 307 6 months Adults with alcohol and drug dependence Patients received treatment through an integrated model, in which primary health care was included within the addiction treatment program Primary care within substance abuse program Examine differences in treatment outcomes and costs between integrated and independent models of medical and substance abuse care Outcomes: 1) alcohol and drug abstinence rate, 2) healthcare utilization; Cost: inpatient, outpatient and treatment costs
Weeks et al. (2009) USA Cross-sectional Cost-comparison Health care payer 63,647 677,901 NA Individuals 65 years and older under Medicare The intervention group was assignment to a large multispecialty group practice in accountable care organizations. Each beneficiary was assigned to a unique primary care physician for a 2-year period. Multispecialty primary care group practice Compare the costs and quality of care provided to Medicare beneficiaries by physicians who worked within large multispecialty physician group practices Outcomes: 1) outpatient clinical measures, 2) ambulatory care-sensitive hospitalisations; Cost: inpatient, long-term and home care
Weaver et al. (2009) USA Cluster RCT Cost-consequence Health care payer 232 199 12 months Individuals with HIV, mental illness and substance abuse disorders Integrated HIV primary care, mental health, and substance abuse services among triply diagnosed patients. Outpatient multidisciplinary mental health, substance abuse and case management services Evaluate the cost-effectiveness of integrating HIV primary care, mental health, and substance abuse services amongst triply diagnosed patients Outcomes: 1) quality of life, 2) mental health scores; Cost: 1) inpatient, outpatient, rehabilitation, home care, alternative, primary care, long-term care; 2) out-of-pocket expenses
Van Orden et al. (2009) The Netherlands Cluster RCT Cost-consequence Health care payer 102 63 12 months Adults with mental illness Patients with mental illness were assigned to collaborative care program in a primary care setting through traditional referral of patients to mental health services Primary care and specialized mental health care Compare the effect of introducing collaborative care on the attached mental health professional model in a primary care setting Outcomes: 1) quality of life, 2) satisfaction with care, 3) mental health score; Costs: treatment costs
Olsson et al. (2009) Sweden Pre-post cohort Cost-effectiveness Health care payer 56 56 18 months Community dwelling older adults 65 years and older, with hip fracture Patient centered integrated care pathway for patients admitted with hip fracture. Multidisciplinary orthopedic hospital ward Compare costs and consequences of integrated care pathways for patients admitted with acute hip fractures Outcomes: 1) activities of daily living score; Cost: 1) intervention and operational costs, 2) implementation costs, 3) inpatient costs
Leeuwen et al. (2015) The Netherlands Cluster RCT Cost-utility Societal perspective 456 691 24 months Community dwelling older adults with frailty The Geriatric Care Model combined regularly scheduled in-home comprehensive geriatric assessments by practice nurses followed by a customized care plan management and training of practice nurses by a regional geriatric expert team Multidisciplinary geriatric primary care team Evaluate the cost-effectiveness of the Geriatric Care Model compared to usual primary care Outcomes: 1) quality adjusted life years, 2) activities of daily living; Costs:1) inpatient, primary, outpatient, home and long-term care & medication costs, 2) informal care giver costs
Lanzeta et al. (2016) Spain Cluster RCT Cost-utility Health care payer 70 70 12 months Individuals with multimorbidity An integrated healthcare model comprising an assigned internist and a hospital liaison nurse for patients with multimorbidity Primary and hospital-based care Examine the effectiveness of an integrated model for patients with multimorbidity, based on an assigned internist and a hospital liaison nurse Outcomes: 1) quality adjusted life years, 2) health resource utilization; Costs: 1) acute, specialists, primary and home care, 2) treatment costs
Goorden et al. (2013) The Netherlands RCT Cost-utility Societal perspective 65 61 12 months Employees sick-listed due to major depressive disorder Collaborative care for major depressive disorder in an occupational healthcare setting Occupational health setting and consulting specialist care Evaluate the cost-utility of a collaborative care intervention in sick-listed employees with major depressive disorder Outcomes: 1) quality adjusted life years, 2) health care utilization; Costs: 1) primary, specialist care and intervention costs, 2) productivity loss
Boland et al. (2015) The Netherlands Cluster RCT Cost-utility Societal perspective 554 532 24 months Patients with chronic obstructive pulmonary disease A multidisciplinary team was trained in motivational interviewing, setting up individual care plans, exacerbation management, implementing clinical guidelines and redesigning the care process Multidisciplinary primary care teams Examine the cost-effectiveness of a disease management program for patients living with chronic obstructive pulmonary disease Outcomes: 1) quality adjusted life years, 2) symptom improvement; Costs: 1) acute, primary, rehabilitation and home care, 2) productivity loss and travel costs
Donohue et al. (2014) USA RCT Cost-utility Health care payer 150 152 12 months Patients with depression following coronary artery bypass graft (CABG) surgery Patients who screened positive for depression after CABG surgery received an 8-month centralized, nurse-provided and telephone-delivered CC intervention for depression Primary care and specialized outpatient mental health care Examine the impact of telephone-delivered collaborative care of treating post-CABG surgery depression compared to usual care Outcomes: 1) quality adjusted life years, 2) depression free days; Costs: acute and outpatient costs
Cohen et al. (2012) Canada Pre-post cohort Cost-consequence Societal perspective 81 Self-comparator 12 months Children with medically complex chronic conditions Clinics at two community hospitals distant from tertiary care were staffed by local community pediatricians with the tertiary care center nurse practitioner and linked with primary care providers Outpatient clinics within community-based hospitals with pediatricians, linked with primary care Evaluate the effectiveness of a community–based complex care clinic integrated with a tertiary care facility Outcomes: 1) health related quality of life; 2) perceptions of care; Cost; 1) inpatient, primary, outpatient and home care. 2) Out-of-pocket expenses for health and social care
Wise et al. (2006) USA Cohort Cost-comparison Health care payer 2010 30,360 12 months Older adults, 65 years and older, with chronic conditions A prospective health risk assessment, point-of-care information management, clinical decision support, multidisciplinary clinical oversight, and a clinical “Health Navigator” to deliver integrated health care Multidisciplinary primary care teams Assess the impact of an integrated set of care coordination tools and chronic disease management interventions on utilization and cost Cost: adjusted acute, primary care and drug costs
McCall et al. (2010) USA Cohort Cost-consequence Health care payer 2,619 2490 36 months Older adults, aged 65 and older who are high cost Medicare users The intervention was a practice-based care management services to high-cost Medicare beneficiaries. Case managers, who were assigned to each physician office, developed relationships with program participants to provide support across the continuum of care Multidisciplinary colocated primary care teams with linkages to home and long-term care Evaluate whether the Massachusetts General Hospital and its case management program can meet targeted cost-savings compared to control Outcomes: 1) comorbidity score, 2) care experience and satisfaction; 3) healthcare utilization; Cost: 1) covered inpatient, primary, outpatient and home care
Simon et al. (2001) USA RCT Cost-effectiveness Health care payer 110 109 6 months Primary care patients with major depressive episode Stepped collaborative care for patients with persistent depressive symptoms after usual primary care management. Patients received collaborative care with liaison psychiatrist and primary care physician. Large primary care clinics part of a health cooperative Evaluate the incremental cost-effectiveness of stepped collaborative care for patients with persistent depressive symptoms after usual primary care management Outcome: 1) depression-free days; Costs: outpatient, primary, specialists and inpatient care
Hebert et al. (2008) Canada Pre-post cohort (D-in-D) Cost-consequence Health care payer 501 419 48 months Older adults aged 65 living with frailty and disability Integrated Service Delivery System developed to improve continuity and increase the efficacy and efficiency of services, especially for older and disabled populations Population level health and social care including: acute, home, long term, rehab and social services Evaluate the impacts of integrated care model for older adults on the use of services and on costs in the experimental zone, compared with the comparison zone Outcomes: 1) functional and mental health scores; 2) care satisfaction; 3) care giver burden; Costs: 1) implementation and operation costs, 2) primary, specialist, acute and outpatient costs
Vroomen et al. (2012) The Netherlands Cluster RCT Cost-utility Societal perspective 201 136 6 months Older adults living in residential homes The intervention consisted of quarterly in-home assessment of residents, multidisciplinary team meetings with primary care physicians, nurse and physiotherapists, and multidisciplinary consultations multidisciplinary residential home care linked with primary care Evaluate the cost-effectiveness of a multidisciplinary integrated care in residential homes Outcomes: 1) quality adjusted life years, 2) functional status, 3) quality of care scores; Costs: 1) acute, primary, outpatient/specialist care, 2) operational/implementation costs; 3) informal caregiver productivity loss
Salmon et al. (2012) USA Pre-post cohort Cost-comparison Health care payer 39,982 Self-comparator 12 months Patients enrolled in collaborative accountable primary care organizations A collaborative accountable care model with registered nurses who served as care coordinators were a central feature of the initiative. They used patient-specific reports and practice performance reports to improve care coordination, identify gaps, and address opportunities for quality improvement primary care physician group practice Examine the impact of accountable coordinated care initiative in three diverse provider practices before and after implementation Outcomes: 1) Outpatient/primary care clinical measures; Cost: 1) Inpatient and primary care, 2) intervention cost
Looman et al. (2016) The Netherlands Pre-post cohort Cost-utility Societal perspective 254 249 12 months Community dwelling older adults with frailty Primary care physician served as the care coordinator and single-entry point for the elderly. Nurse practitioner visited patients for cognitive, mental and functional assessment who also provided case management. A multidisciplinary treatment plan was then developed. Multidisciplinary primary care team linked with nursing home and outpatient/specialist care Examine the impact of integrated model for community-dwelling older adults with frailty Outcome: 1) quality adjusted life years; Cost: 1) inpatient, primary, home, outpatient and nursing home care, 2) intervention operational costs, 3) Informal care giver costs
Celano et al. (2016) USA RCT Cost-utility Health care payer 92 91 6 months Patients hospitalized for cardiovascular illness with mental illness Psychiatric treatment in the intervention was provided in concert with the patients’ primary medical clinicians—within a framework supervised by a psychiatrist Inpatient care followed by with telephone outpatient follow up, with primary care linkages Examine the cost-effectiveness and differences in healthcare utilization and cost between collaborative depression and anxiety program with usual care Outcomes: 1) quality adjusted life years, 2) mental health status; Costs: 1) acute, primary, outpatient/specialist care
Markle-Reid et al. (2010) Canada RCT Cost-consequence Societal perspective 55 54 6 months Community dwelling older adults, 75 years and older, at risk for falls A six-month multifactorial and evidence based falls prevention strategy involving a multidisciplinary team Multidiscplinary home care linked with primary care and community services Determine the effects and costs of a multifactorial, multidisciplinary team approach to falls prevention compared with usual home care services. Outcomes: 1) falls; 2) clinical outcomes (functional, mental and cognitive scores); 3) quality adjusted life years; Cost: 1) Acute, home, primary and community care, 2) out-of-pocket indirect medical expenses
Pozzilli et al. (2002) Italy RCT Cost-consequence Health care payer 133 68 12 months Patients diagnosed with multiple sclerosis The home-based multidisciplinary team collaborated with the patient, physician, and caregiver in designing individualized clinical care and in coordinating home services with hospital care Multidisciplinary home care with specialists linkages Compare the effectiveness and the costs of multidisciplinary home-based care in multiple sclerosis with hospital care Outcomes: 1) quality of Life, 2) health resource utilization; Costs: 1) inpatient, outpatient and home care, 2) intervention costs
Tzeng et al. (2007) China RCT Cost-consequence Health care payer 257 247 6 months Individuals diagnosed with schizophrenia A network of mental health services was created by coordinating a general acute care hospital, a day hospital, a psychiatric rehabilitation institution, a community rehabilitation center, home visit providers, a specialized psychiatric hospital, and local clinics Network of acute care, day hospital, rehabilitation, home care providers and local clinics Compare the cost-effectiveness of an integrated model of schizophrenia treatment with those of the traditional treatment model provided by acute care Outcomes: 1) quality of life, 2) care giver burden, 3) health service utilization, Costs: inpatient, outpatient, rehabilitation and home care
Bergmann et al. (2017) Malawi and Mozambique Pre-post cohort (D-in-D) Cost-effectiveness Health care payer Not reported Not reported 24 months Children under 5 years with HIV who were underweight Integration of health and nutrition program areas identified as important in reducing the vulnerability of children impacted and infected by HIV/AIDS: infant and young child feeding, prevention of mother-to-child transmission of HIV, pediatric HIV care and treatment, and community-based management of acute malnutrition Community based health workers and community clinics for HIV and acute under-nutrition To estimate the impact and cost-effectiveness for integrated HIV and nutrition service delivery in sub-Saharan Africa Outcomes: 1) HIV infections averted, 2) undernutrition cases cured, 3) disability adjusted life years; Cost: 1) intervention costs, 2) operational/implementation costs, 3) life long HIV treatment cost
Koch et al. (2017) USA Pre-post cohort (D-in-D) Cost-comparison Health care payer 2.5 million per year 2.5 million per year (their own control) 15 months Patients served by hospital and physician groups merged as part of horizontal integration of care Vertical integration through a set of physician acquisitions by hospital systems Hospital, primary and specialist care physicians Assess how (financial) vertical integration affects volume and cost of services provided by acquired physicians and hospitals Outcome: health care utilization; Cost: acute, primary and outpatient/specialist care
Rosenheck et al. (2016) USA Cluster RCT Cost-utility Health care payer 223 181 24 months Individuals aged 15-40 in treatment for first episode of psychosis A multidisciplinary, team-based treatment approach for first episode psychosis. This included: personalized medication management, family psychoeducation, individual, resilience-focused illness self-management therapy, and supported education and employment Multidisciplinary community mental health treatment clinics Compare the cost-effectiveness of a comprehensive, multidisciplinary, team-based treatment approach for first episode psychosis to usual community care Outcomes: 1) quality adjusted life years, 2) health service utilization; Costs: 1) inpatient, outpatient, residential and nursing home care and medication costs, 2) implementation and operational costs
Sahlen et al. (2016) Sweden RCT Cost-utility Health care payer 36 36 6 months Patients diagnosed with congestive heart failure The patients were offered structured person-centered palliative care at home with easy access to care, and the team was responsible for the total care, including co-morbidities Multidisciplinary palliative home care team linked with specialist care To assess the cost-effectiveness of person-centered integrated heart failure and palliative home care Outcomes: 1) quality adjusted life years; Costs: 1) acute, home, primary and specialist care, 2) intervention cost
Blom et al. (2016) Netherlands RCT Cost-consequence Societal perspective 3145 4133 12 months Community-dwelling older adults, 75 and older living with complexity The general practitioner (GP) or the practice nurse (under supervision of the GP) made an integrated care plan for participants with complex problems. Other care professionals were involved where needed through multidisciplinary consultations General Practice with geriatric assessment training Assess the effectiveness and cost- effectiveness of a monitoring system to detect the deterioration in somatic, functional, mental or social health Outcomes: 1) quality of life, 2) activities of daily living, 3) satisfaction with care 4) Informal care giver time; Cost: 1) acute, primary, outpatient, nursing home and medication 2) intervention costs, 3) implementation costs, 4) informal care costs
Pimperl et al. (2017) Germany Pre-post cohort Cost-consequence Health care payer 5411 5411 48 months Individuals enrolled with the accountable care organization insurance scheme Accountable care organization focused on population health management with a Triple Aim framework Cross-sectoral cooperation of physicians, hospitals, social care, nursing staff, therapists, and pharmacies Identify an appropriate study design for evaluating population health outcomes of accountable care organization such as based on shared savings contract Outcomes: 1) survival, 2) comorbidity score; 3) Costs: outpatient physician and specialist care, hospital, rehabilitation, medication costs
Schellenberg et al. (2004) Tanzania Cohort Cost-consequence Societal perspective 100,000 100,000 24 months Children with malaria, pneumonia, malnutrition and diarrhea The intervention was designed to increase children’s survival at household, community, and referral levels, with three components: improvements in case-management, improvements in health systems, and improvements in family and community practices Family and community primary care practices and hospitals Assess the effectiveness of facility- based integrated management of childhood illness in rural Tanzania Outcomes: 1) child health outcomes, 2) household health behavior 3) children’s mortality; Costs: 1) drugs and vaccines, 2) implementation costs & operational costs, 3) intervention costs, 4) out-of-pocket expenses, 5) acute, primary and community care costs
Bird et al. (2012) Australia Pre-post cohort Cost-consequence Health care payer 223 Self-comparator 36 months Children with asthma that presented frequently at the emergency department Patients received care facilitators who provided assistance in the promotion of carer/self-management, education and linkage to an integrated healthcare system, comprising of acute and community-based healthcare providers. Acute, primary and other community-based care Assess a model of care for pediatric asthma patients aimed to promote health and reduce their acute care utilizations Outcomes: 1) activity limitation and emotional function, 2) acute care utilizations; Cost: 1) intervention costs; 2) acute care costs
Goltz et al. (2013) Germany Cohort Cost-consequence Health care payer 2455 2455 36 months Patients with osteoporosis who experienced index fractures Patients received multidisciplinary cooperation between different sectors of the health care system, improved diagnostics, optimized drug therapy as well as encouraging lifestyle changes such as adequate nutrition and exercise. Ambulatory care Evaluate the outcomes of patients participating in a program of integrated care for osteoporosis in terms of medication supply, fracture incidence and expenses Outcomes: 1) fracture Incidence, 2) occurrence of pain; Costs: 1) acute care costs, 2) treatment costs, 3) medication costs
Steuten et al. (2007) Netherlands pre-post cohort Cost-utility Societal perspective 2455 2455 60 months Patients 18 years and older with GP diagnosis of asthma Care was delivered by a collaborative practice team consisting of a pulmonologist, primary care physician, and respiratory nurse specialists. The respiratory nurse specialists act as liaison between primary and secondary care primary care collaborating with specialists Assess long-term cost-utility of a disease management program for adults with asthma was assessed compared to usual care Outcomes: 1) quality adjusted life years, 2) asthma related exacerbations/control; costs: 1) acute, primary and outpatient/specialist care costs, 2) medication costs 3) treatment costs, 4) implementation and operational costs, 5) patient productivity loss
Wiley-Exley et al. (2009) USA Cluster RCT Cost-utility Societal perspective 1257 1948 6 months Older adults, 65 years and older with major depressive disorder in primary care Patients required referral to a behavioral health provider outside the primary care setting, and the behavioral health provider had primary responsibility for the mental health/substance abuse needs of the patient Multidisciplinary specialist team colocated in primary care Compare the cost-effectiveness of integrated care in primary care to enhanced specialty referral for older adults with behavioral health disorders Outcomes: 1) depression free days, 2) quality adjusted life years; Costs: 1) inpatient, emergency room use, nursing home, rehabilitation care. 2) medication costs, 3) caregiver and patient indirect costs (transportation, productivity loss)
Karow et al. (2012) Germany Cohort Cost-utility Health care payer 64 56 12 months Adults patients diagnosed with first or multiple-episode schizophrenia Patients receive a multidisciplinary team with a small client/staff ratio, home-treatment, high-frequent treatment contacts, no dropout policy and 24-hour availability Inpatient, outpatient/specialists and occupational therapy care To compare the cost effectiveness of therapeutic assertive community treatment with standard care in schizophrenia. Outcomes: 1) quality adjusted life years, Costs: 1) inpatient care, day-clinic care, outpatient and specialist care costs, 2) medication costs
Renaud et al. (2009) Burundi Cohort Cost-effectiveness Health care payer 149 Self-comparator 60 months People living with HIV who initiated antiretroviral treatment Care was given in primary health care facilities, which favours a more personal link with patients. Secondly, these health facilities delivered integrated care for people living with HIV. Primary care based on non-for-profit organization delivering care for individuals living with HIV Calculate the incremental cost effectiveness of an integrated care package for people living with HIV/AIDS in a not-for-profit primary health care centre. Outcomes: 1) disability adjusted life years; Cost: 1) outpatient, acute and home care costs 2) medication costs 3) intervention costs, 4) food support costs
Tanajewski et al. (2015) United Kingdom RCT Cost-utility Health care payer 205 212 3 months Older people at risk of adverse outcomes after acute care discharge This intervention comprised geriatrician assessment of patients on the acute medical unit and further short-term community follow-up to continue the assessment and oversight of the delivery of medical and non-medical community interventions Multidisciplinary acute care team and links to primary care To examine the cost-effectiveness of a specialist geriatric medical intervention for frail older people in the 90 days following discharge from an acute medical unit Outcomes: 1) quality adjusted life years; Costs: 1) acute, primary and specialist care; 2) intervention costs
Lambeek et al. (2010) Netherlands RCT Cost-utility Health care payer 66 68 12 months Individuals visiting outpatient clinic due to low back pain Integrated care consisted of a workplace intervention based on participatory ergonomics, with involvement of a supervisor, and a graded activity programme based on cognitive behavioural principles Occupational health setting linked with multidisciplinary outpatient team To evaluate the cost effectiveness of an integrated occupational health programme for sick listed patients with chronic low back pain Outcomes: 1) duration until sustainable return to work; 2) quality adjusted life years; Costs: 1) primary and secondary care, home care, and drugs. 2) Out of pocket expenses for additional and informal care; 3) patient productivity loss
Bertelsen et al. (2017) Denmark RCT Cost-utility Societal perspective 106 106 12 months Adult patients admitted to the hospital with acute coronary syndrome A model of shared care cardiac rehabilitation that included general practitioners and the municipality Shared care between primary care and outpatient public health centers, with multidisciplinary teams To assess the cost-utility of shared care cardiac rehabilitation versus hospital-only cardiac rehabilitation from a societal perspective Outcomes: 1) quality-adjusted life years; Costs: 1) intervention cost/formal and informal staff time; 2) primary and secondary care; 3) productivity loss
Camacho et al. (2018) United Kingdom Cluster RCT Cost-utility Health care payer 191 196 24 months Patients with depressive symptoms and a record of diabetes and/or coronary heart disease Participants attending primary care physician practices allocated to the collaborative care group received up to eight face-to-face sessions of brief psychological therapy delivered by a case manager over 3 months General practices with case managers co-located with multidisciplinary team To assess the cost-effectiveness of collaborative care for people with depression in the context of multimorbidity Outcomes: 1) depression severity; 2) quality adjusted life years; 3) health care utilization; Costs: 1) inpatient; 2) outpatient; 3) emergency; 4) primary/community care; 5) intervention costs; 6) Implementation and training costs
Everink et al. (2018) The Netherlands Cohort Cost-utility Societal perspective 113 49 9 months Community-dwelling older patients who were admitted to a geriatric rehabilitation facility The integrated care pathway comprised of cross-organizational agreements on coordination and continuity of care for older patients who transfer between the hospital, the geriatric rehabilitation facility and primary aftercare in the home context Coordination between the hospital, the geriatric rehabilitation and primary care and home care To determine the cost-effectiveness of receiving usual care compared to receiving care in the integrated care pathway Outcomes: 1) dependence in activities of daily living; 2) quality adjusted life years; Cost: 1) intervention costs; 2) implementation costs; 3) primary, home care, long-term care, acute care and allied professionals; 4) patient out-of pocket expenses; 5) informal caregiving
Wong et al. (2018) China RCT Cost-utility Health care payer 43 41 24 months End-stage heart failure patients referred to in-hospital palliative care services Patients received a transitional homebased palliative end-stage heart failure program delivered by nurse case managers who were trained specialist palliative care home care nurses with experience in heart failure management Transitional care between hospital to home care delivered by case manager and multidisciplinary home care team To evaluate the cost-effectiveness of a transitional home-based palliative care program Outcomes: 1) quality adjusted life years; Costs: 1) acute, home and emergency care; 2) intervention cost; 3) training cost
Uittenbroek et al. (2018) The Netherlands RCT Cost-utility Societal perspective 747 709 12 months Older adults, aged 75 and over with primary care providers A primary care physician-led Elderly Care Team was assembled for each participating practice, which also consisted of an elderly care physician, a community nurse, and a social worker General practitioner-led elderly multidisciplinary care team in primary care with case manager To assess the cost-effectiveness of integrated geriatric care team in primary care Outcomes: 1) quality adjusted life years; 2) number of days older adult was able to age in place (i.e. no nursing home stays); Costs: 1) primary, acute, medication and paramedical care; 3) social and home care; 4) informal caregiving
Tsiachristas et al. (2015) The Netherlands Cohort Cost-utility Societal perspective 1034 1034 24 months patients diagnosed with or at risk of cardiovascular disease and chronic obstructive pulmonary disorder Programs focused on improving the collaboration between different disciplines of health care professionals and redesigning the care giving process toward proactive, patient-centered care. Disease management programs implemented through collaborations between general practices and hospitals, primary care practices (including physiotherapists and dieticians), or primary and community settings To evaluate the cost-effectiveness of disease management programs for patients diagnosed with or at risk of cardiovascular disease and chronic obstructive pulmonary disorder Outcomes: 1) quality adjusted life years; 2) level of physical activity; 3) proportion of smokers; Costs: 1) health care utilization costs; 2) travel costs; 3) productivity loss; 4) development costs; 5) implementation costs

Quality assessment

Study design

Randomized control trials (58%) were the most commonly used study designs, followed by observational cohort designs (41%), where 16% of the studies were quasi-experimental or used pre-post cohort designs. Only one study was cross-sectional in nature.

To minimize bias while conducting the economic evaluations, 91% of the studies identified a comparator or control group, which did not receive the intervention. However, the baseline population characteristics in the groups were significantly different in almost a third of the studies, particularly in observational cohort designs. Quasi-experimental approaches such as difference-in-difference analysis of the pre-post intervention period were conducted in 7% of the studies. This was performed to ensure that the evaluations were in fact measuring the impact of the intervention rather than other confounding factors that might bias the results. Authors such as Pimperl et al. used propensity-score matching to ensure individuals enrolled in an accountable care organization were similar at baseline to their comparator group on socio-demographic and clinical variables [63]. Majority of the longitudinal studies collected data at multiple points, spanning beyond baseline and follow-up (61%). Furthermore, 70% had clear description of patient attrition or drop-outs at the follow-up period. Contamination due to the exposure of individuals in the comparator to the intervention, occurred in a third of the studies. To avoid this bias, 20% of the studies used a cluster RCT design, where individuals were randomized at the care setting level, rather than at the patient level.

Measurement of cost and outcomes

Most studies explicitly stated the perspective adopted in their economic evaluation (75%), with the healthcare payer perspective dominating the broader societal perspective (66% vs. 34%). Studies that adopted a societal perspective also considered the indirect impact of the intervention on caregiver burden, out of pocket care expenses and productivity loss. Non-medical and indirect costs were considered in 35% of the studies. For example, van Leeuwen et al. valued the indirect costs of a multidisciplinary geriatric primary care team on the informal care of frail elderly patients [31]. Only a third of the studies included costs associated with the development of the intervention or the implementation costs, with the majority only considering intervention and healthcare utilization costs. Healthcare costs and utilization from across all relevant health and social sectors were reflected in 66% of the studies. However, while some studies reflected both types of information, others only reported overall health care costs without the resource utilization. For example, in a cross-sectional study of a large multispecialty primary care group practice in the US serving Medicare recipients, despite including all relevant health care costs (such as home healthcare, long-term care, skilled nursing facilities and acute care), the resource utilization associated with each sector was not reported [68]. The follow-up period varied across studies, from short time horizons such as 3 months [34], to up to 5 years [49, 67]. Those that measured the impact of the interventions within a year did not require discounting of costs and health benefits (53%) because they were expressed in present values [73]. However, only 18% of the studies with a time horizon longer than a year applied discounting.

Statistical analysis and presentation of data

Full economic evaluations of integrated care interventions applied the incremental cost-effectiveness ratio analysis (ICER) in their approach (55%), rather than using the net monetary (NMB) or health benefit (NHB) (7%). Studies that reported the ICER and NHB estimated the joint monetary and effect differences between the intervention and a comparator. Various approaches were adopted to address uncertainty around the reported cost-effectiveness estimates, including presenting both cost-effectiveness acceptability curves and cost-effectiveness planes. This involved demonstrating whether the integrated care intervention met or surpassed society’s willingness to pay (WTP) for an additional unit of health benefit (43%). For example, in an evaluation of a community-based intervention for frail older adults, Looman et al. graphically reported their ICERs on a cost-effectiveness plane [46]. Using this approach, they demonstrated that compared to usual care, the new intervention was only 0.21% less costly and more effective, while 78.8% more costly and less effective. Because decision-criteria such as WTP thresholds or cut-off points were not applied to the results, it was challenging to determine whether this intervention could be conclusively deemed cost-effective [46]. On the other hand, Tanajewski et al determined that a multidisciplinary discharge program was more favorable than usual care using the NMB analysis at various WTP thresholds (£20,000–120,000). They also showed that the more decision-makers were willing to invest in the intervention, the higher the probability of its cost-effectiveness [34]. Subgroup analysis was used to examine the heterogeneity of economic impact or the source of variability, but was conducted in 47% of the studies in this review for more information, please refer to Table 2 for the checklist assessing the quality of economic evaluations.

Table 2

Checklist assessing the quality of economic evaluations.

Category Item Description Zulman et al. (2017) Weisner et al. (2001) Weeks et al. (2009) Weaver et al. (2009) Van Orden et al. (2009) Olsson et al. (2009) Leeuwen et al. (2015) Lanzeta et al. (2016) Goorden et al. (2013) Boland et al. (2015) Donohue et al. (2014) Cohen et al. (2012) Wise et al. (2006) McCall et al. (2010) Simon et al. (2001) Hebert et al. (2008) Vroomen et al. (2012) Salmon et al. (2012) Looman et al. (2016) Celano et al. (2016) Markle-Reid et al. (2010) Pozzilli et al. (2002) Tzeng et al. (2007) Bergmann et al. (2017) Koch et al. (2017) Rosenheck et al. (2016) Sahlen et al. (2016) Blom et al. (2016) Pimperl et al. (2017) Schellenberg et al. (2004) Bird et al. (2012) Goltz et al. (2013) Steuten et al. (2007) Wiley-Exley et al. (2009) Karow et al. (2012) Renaud et al. (2009) Tanajewski et al. (2015) Lambeek et al. (2010) Bertelsen et al. (2017) Camacho et al. (2018) Everink et al. (2018) Kam et al. (2018) Ulttenbroek et al. (2018) Tsiachristas et al. (2015) %

Study design 1. Design was experimental (e.g. RCT or cluster-RCT) or quasi experimental design (e.g. used propensity score matching, pretest-posttest design)? 75
2. Random allocation into intervention and control groups 57
3. The study population consist of an intervention and control group Y 91
4. Relevant baseline characteristics are comparable NA 72
5. The interventions or strategies being compared are described 98
6. Included more than just baseline and follow up period 61
7. Clear description of inclusion and exclusion 86
8. Clear description of drop-outs 70
Intervention setting 9. Stated relevant aspects of the system(s) in which intervention takes place 84
10. Co-interventions or contamination are avoided 32
Measurement of costs & outcomes 11. Describe the perspective of the study and relate this to the outcomes and costs being evaluated. 75
12. Described which outcomes were used as the measure(s) of benefit in the evaluation 100
13. Inclusion of development and implementation cost 34
14. Inclusion of cost & utilization from across all relevant health and social sectors 66
15. Inclusion of direct non-medical and indirect costs 39
16. Justification for omitting costs categories N/A NA NA NA NA NA 29
17. The sources of resource utilization and cost are described 91
18. The resource utilization and costs are reported separately 75
19. Reports the (adjusted) dates of estimated resource quantities and unit costs Y Y Y Y Y 73
20. Discounting of outcomes and costs performed NA NA Y NA NA NA NA Y NA NA NA NA NA NA NA NA Y NA NA Y NA N/A NA NA NA NA 18
Statistical analysis 21. Data analysis is performed according intention-to-treat principle 39
22. Dealt adequately with missing observations 59
23. Appropriate statistical methods for analysing skewed data 68
24. Report the values, ranges, references, and if used, probability distributions for all parameters. 91
25. Analysed cost-effectiveness using the incremental cost- effectiveness ratio (ICER) method 55
26. Analysed cost-effectiveness using the incremental net- monetary or health benefit (INB) regression method 7
27. Performed sub group analysis to examine heterogeneity of results 48
28. Analysed the uncertainty in the estimates of the costs and effects 64
Presentation of data 29. A decision criteria is applied to determine whether to reject or accept intervention (e.g. willingness-to-pay vs. cost effectiveness threshold) 43
30. The study discusses the generalizability of the results to other context and/or patient groups 98

Legend: yes ✓, no ○.

Emerging Challenges

Time horizon

Limitation associated with the study timeframe was the most frequently cited barrier to conducting robust evaluations of integrated care interventions. In a cost-effectiveness analysis of a multidisciplinary residential care for frail older adults, Vroomen et al. note that: “the [six months] duration of the trial was relatively short because of the high risk for drop out owing to the extreme vulnerability of residents” [45, p. 5]. A further limitation of the study’s internal validity was the patients’ maturation effect: “patients in a residential home have a heterogeneous mix of chronic conditions that naturally erode over time which makes it difficult to know if an intervention would be able to override the downward trend…in such a short time span” [45, p. 6]. Other authors questioned whether measures such as mortality or quality adjusted life years (QALYs), often the standard in economic evaluations, were sensitive enough to truly capture the effect of the intervention in short follow-up periods [45, 63].

Underestimation of the downstream health and monetary benefits of integrated care interventions was also a significant concern in studies with short time horizons [42]. In a 12-month cost-utility analysis of a collaborative care program targeting patients with depression and cardiovascular illness, Donohue et al. highlight that in this type of intervention, “most cost savings [occur] between the first and second years of follow-up, stressing the continued need for adequately powered and longer-term trials” [44, p. 457]. Zulman et al. argue that when determining an appropriate follow-up period, it is important to not only account for the implementation period but also the potential initial intensity of patients care needs: “once patients are enrolled in such programs, it takes time to build their trust, modify health behaviors and improve chronic disease trajectories….which could translate to increased [initial need for] health maintenance and screening, but subsequent reductions in future utilizations” [50, p. 172].

Finding suitable comparators

Authors acknowledged the challenge of finding a comparator population that could serve as an appropriate control for those receiving the intervention. Observational designs were often adopted because of the difficulty in identifying participants for a comparator group. Olsson et al. note that it would have been ideal to randomize community-dwelling older adults to usual care vs. a multidisciplinary geriatric primary care team. However, it was impractical due to the administrative burden: “there were concerns about the difficulty for nursing staff working in two care systems at the same time and for patients possibly comparing the treatment they received “[64, p. 1633]. In many observational studies, the significant baseline differences between those receiving the intervention and usual care emerged as a major threat to the studies’ validity. In an integrated intervention targeting high cost Medicare users, Mccal et al highlight that: “…because the comparison group was not based on random sampling….if the intervention had a disproportionate number of high risk, more cost increasing beneficiaries, then the [the evaluation results] could be biased against the intervention” [57, p. 143]. However, even in RCTs, the hidden differences in the context of the comparator and the intervention group could jeopardize the attribution of effect, or causality [15]. To address this potential bias, studies adopted quasi-experimental approaches to adjust for baseline differences between comparison groups. In a cost-effectiveness study of disease management programs in the Netherlands, Tsiachristas et al. note that, “[because] of the observational study design, the patients in the comparators in each disease category differed in disease severity and sociodemographic characteristics at baseline. Therefore, we used propensity score matching to reduce confounding caused by these differences” [40, p. 978].

Risk of contamination bias

Controlling for the potential of contamination bias emerged as a challenge in both observational and non-clustered RCT designs. Often due to feasibility issues, both usual care and the integrated care intervention were delivered in the same care setting, alongside each other. Zulman et al. suggest that in their study, clinicians in the usual care group could have observed and adopted the practices of the intensive case management program in the Veterans Affairs medical care home program [50]. In other studies, this bias was even more difficult to control when plans to spread the integrated care intervention across the regional setting were already underway. This was exemplified by a primary care-based collaborative mental health intervention in the Netherlands. The authors note that: “when the study started, about 65% of the general practitioners in the Hague were already participating in the program” [55, p. 78]. In large scale implementations of integrated care, there is a risk of diluting the true magnitude of impact, as it can prove difficult to confine patients receiving usual care from also accessing these services [63].

Post-hoc evaluation culture

A significant barrier was the implementation of the interventions without planning for an economic evaluation. In a study of a community-based program integrating HIV and nutrition care in Malawi and Mozambique, Bergmann et al. noted that “the project was set up without a research design for rigorous impact evaluation” [66, p. 710]. This limited data accessibility and the suitability of the outcome measures selected for the evaluation: “we would have preferred to directly estimate the effects of the [intervention] on morbidity and mortality but did not have the data to do so” [66, p. 710]. Other observational studies that were not initially planned as experimental study designs had to rely on routinely collected data to estimate the magnitude of impact. For example, claims data from widely accessible electronic information systems were used to measure the impact of a population-based accountable care organization in Germany. While this reduced the intensity of resources required for the evaluation, authors noted that their approach may have led to the underestimation of the broader impact of the intervention, such as its effect on patient and caregiver out-of-pocket spending [63].


Main results

The results of economic evaluations of integrated care should be placed in the broader context of its implementation and the methodological approaches used in its evaluation. To our knowledge this is the first review that critically appraised the methodological quality of economic evaluations of integrated care interventions, against best-practice guidelines. Our review found significant differences in quality. Some studies showed poor methodological rigor, challenging conclusions on the cost-effectiveness of integrated care. Similar to Nolte and Pitchforth’s review of systematic reviews on the economic impacts of integrated care, we found wide variability across study designs, measurements of costs and outcomes, as well as analytical approaches and presentation of results [5].

Some of the key cited challenges to robust economic evaluations in our study were related to: 1) time horizon of the evaluation; 2) inadequate or lack of comparator group; 3) contamination bias due to potential exposure of those in usual care with the treatment; and 4) a post-hoc evaluation culture.

Interpretation of findings in the context of other studies

Several reviews highlighted similar challenges as impediments to arriving to robust evidence on the economic impacts of integrated care. In a review of interventions targeting frequent users of the emergency department, Althaus et al. note that it was difficult to attribute cost savings to the interventions when a significant number of studies did not have comparators [74]. De Bruin et al. highlight that there was a large variation in the types of comparators used in their review of the impact of disease management programs on health expenditures. They found that when there were similarities in the intervention and usual care, it was difficult to observe differences. The ability to contrast between the two groups was further challenged by the poor description of usual care conditions in the majority of the studies [13].

Because integrated care can impact a broad range of costs within and beyond the healthcare system, a broader societal perspective is preferred when estimating costs [15]. In our review, the healthcare perspective dominated the economic evaluations. This could be because it requires much less time and financial resource for data collection. While there is variation in the cost perspectives suggested by national guidelines for health technology assessment, they often also recommend reporting costs from the societal perspective. This is particularly the case if the intervention has an expected impact on other sectors [75, 76]. In a review of collaborative models for individuals with depression, none of the reviewed economic evaluations included costs beyond the healthcare sector [77]. A narrow scope in the cost perspective was also reported in Wong et al.’s systematic review of economic evaluations of integrated care for cardiac rehabilitation. Using the Drummond’s check list to examine the quality of economic evaluations, they found that only 9% of studies included all the relevant costs and consequences for both the intervention and comparator [18]. This narrow scope fails to incorporate the impact of integrated care on other types of care (e.g. Social Services), government departments (e.g. Justice and Education) and productivity levels in the overall economy [32, 78]. An alternative would be to follow the recommendations of the specific country of origin, and to include a societal perspective as a sensitivity analysis.

Best practice recommendations

Guidelines for health technology assessment around the world recommend adopting a life-time horizon [16, 17, 23]. Drummond and colleagues recommend, that in treatment of chronic diseases where benefits may have long-lasting implications, it is often necessary to extrapolate the effects and costs of the interventions being compared over a life time [20]. However, short time horizon was acknowledged as a major limitation in most studies. In our review, 81% of the studies had less than two years follow-up period. Only one study employed decision analytic modelling to extrapolate costs and outcomes estimated during the follow-up period to patient’s lifetime [49]. This approach also allows the linkage of intermediate endpoints (e.g. clinical status) to final endpoints (e.g. QALYs and mortality) [23]. The scarcity of model-based economic evaluations in integrated care maybe due to lack of health economic modelling expertise in this field of research. Another reason could be the complex nature of integrated care, with a non-linear relationship between interventions and outcomes [15]. In addition, delays in the implementation of integrated care interventions may lead the “full” treatment effect to be observed in 3–5 years after the start of implementation [79]. Decision models would not be able to overcome this limitation. Furthermore, evaluations with longer time horizons have an increased risk of the intervention eventually becoming usual care or being contaminated by other initiatives. Tsiachristas et al. suggest setting up routine monitoring of key measures as a potential strategy to measure the long-term effects of the intervention, beyond the research period [15].

Best practice recommendations emphasize the need for a comparator in economic evaluations, often in the form of current practice or variations of similar programs [15], [16, 28]. Our review found that identifying the appropriate comparator can be challenging, particularly for observational studies. To ensure that the evaluation is in fact measuring the causal effect of integrated care, an appropriate control population should be chosen [16, 17]. RCTs are commonly used to minimize differences between the compared populations (i.e. intervention and control arms). However, addressing the differences in observational studies may prove difficult and resource-intensive because the richness of required data and statistical expertise. Furthermore, the risk of contamination by the control group from the intervention is elevated, especially when integrated care implementation takes place in large parts of the population. As such, even cluster-RCTs may be difficult to overcome this risk if not well designed [80]. Quasi-experimental designs provide feasible alternatives in the evaluation of integrated care, when control groups are identified. An approach such as propensity-score matching could reduce observed confounding between the comparators, and is increasingly used in observational studies [81, 82]. Approaches such as difference-in-differences, instrumental variables, and regression discontinuity could reduce the unobserved confounding between the comparators [15, 83].

Integrated care is considered a complex intervention, amenable to being tailored to the context in which it is implemented in. Economic evaluation plays an important role in this complex and adaptive intervention, learning through feedback loops of patients and provider experiences and outcomes [15]. Our review found that economic evaluations in integrated care were often piggy backing on larger scale evaluations. In some instances, the plans for evaluation began after the intervention had concluded [66]. When this is the case, researchers may lack control over the types of outcomes measures included in the routinely collected data [15]. In a qualitative study examining stakeholder perspectives on the evaluation of chronic disease management programmes in six European countries, lack of an evaluation culture was also cited as one the main barriers towards producing sound evidence [84]. There are several potential explanations for this. Policy makers and practitioners maybe unaware of the need for, or benefits of conducting evaluations [84]. There may be a reluctance by stakeholders to support evaluations due to perceived additional administrative and financial burdens [64]. Finally, in some health care contexts, particularly in low-resource settings, there might be a lack of personnel capacity or necessary skill sets to undertake comprehensive economic evaluations [85]. Nonetheless, with the increased reliance on evidence generated from empirical studies by decision-makers, it is important for evaluators to be embedded early, from the design and planning of the integrated care programme, rather than post-implementation [15].

Strengths and Limitations

The extensive search strategy used to capture the concepts of integrated care and economic evaluation is a major strength of this review. A second strength is that our checklist was adapted from the CHEERS and the HTA-DM quality assessment guidelines, which were developed by experts in the field for the purpose of optimizing the reporting of evaluations and health economic evaluations [17, 28]. However, this review should be interpreted in the context of several limitations. Integrated care as a concept has been widely used to achieve various objectives, which explains why there remains a lack of common definition which is universally accepted [86, 87, 88]. While our search strategy attempted to include studies that broadly fit with in our definition, our review may have missed others. Rather than presenting the review as a compendium of all economic evaluations conducted in integrated care, we hope this provides a snapshot into the current practices in the field. Secondly, the checklists provided a guiding framework for critically reviewing the economic evaluations reported by the articles [17]. However, there was room for subjective interpretation which may have biased the scoring. We attempted to address this bias through two reviewers independently appraising the articles against the checklist, with disagreements resolved through reaching consensus.

Implications for research, policy and practice

Economic evaluation cannot be viewed as separate from programme evaluation. Rather, it needs to become an integral part of any evaluation effort [15]. Given the wide implementation of integrated care as a viable approach to tackling complex health and social needs, it is necessary for researchers to step up their efforts in understanding how exactly integrated care works and which transferable lessons can be drawn [88]. This necessitates a multidisciplinary approach in research, where economic evaluation forms a crucial part of a mixed methods approach. Recent developments in applying realist evaluation and multi-criteria decision analysis (MCDA) as part of implementation research in Europe and Australia show promising results [89, 90]. However, they also highlight the complexity and resource intensity necessary to evaluate integrated care. For policy makers to create enabling conditions, researchers also need to become more engaged in the promotion of an evaluation culture. Routinely reporting on short-, medium- and long-term outcomes may help foster a better understanding among policy makers on adequate time horizons necessary for evaluation. Policy makers also need to provide financial and regulatory frameworks for research to become an integral part of designing integrated care initiative, similar to the Innovation Fund in Germany. More importantly, evaluation needs to be viewed as an integral part of integrated care implementation, as a powerful tool to promote cultural change [88]. This is where practice needs to accept that (economic) evaluation is not a means to punish or cut resources, but can be used to support quality improvement and system change. In a complex environment, regular on-going feedback and monitoring are prerequisites to reaching better outcomes [88]. Given the continuing resource restrictions, alongside the inadequate and ineffective use of resources in health and social care, economic evaluation should be high on any priority list, in research, policy and practice.


Arguably a key challenge to the evaluation of integrated care lies in the complexity of the intervention and the short-term evaluation period. This can make it significantly more difficult to perform rigorous economic evaluations; especially when health economists are not involved in the study design. However, methodological gaps in economic evaluations may be more straightforward to address than cultural barriers to evaluation [20]. Therefore, it is important for evaluators to use best-practice standards when planning and conducting economic evaluations. Work by authors such as Drummond et al. has sought to specify the essential components of good economic evaluations in health care and to develop checklists that can help researchers and practitioners critically appraise the methodological quality of studies [17, 19, 20]. To build a reliable evidence base for decision-makers and practitioners utilizing evidence from integrated care studies, it is important to determine the mechanisms that influence the economic impacts of integrated care. This will require designing studies that can reliably answer: is this intervention cost-effective, using which resources, in which settings and for whom? [15] With the increased interest in integrated care from policy makers, the reporting of economic evaluations should be further standardized to allow transferability and transparency [20]. Thorough and reliable economic evaluation should be an integral part of informing the decision-making of integrated care implementation.

Additional File

The additional file for this article can be found as follows:

Appendix 1

Search terms. DOI:


Dr. Willemijn Looman, Erasmus School of Health Policy & Management, Erasmus University Rotterdam, the Netherlands.

Dr Alexander Pimperl, Vice Chairman of the Board, OptiMedis AG, Germany.

Competing Interests

The authors have no competing interests to declare.


  1. Goodwin, N, Dixon, A, Anderson, G and Wodchis, W. Providing integrated care for older people with complex needs: lessons from seven international case studies; 2014. London: King’s Fund. 

  2. Berwick, DM and Hackbarth, AD. Eliminating waste in US health care. Jama, 2012; 307(14): 1513–1516. DOI: 

  3. Hildebrandt, H, Schulte, T and Stunder, B. Triple aim in Kinzigtal, Germany: improving population health, integrating health care and reducing costs of care–lessons for the UK? Journal of Integrated Care, 2012; 20(4): 205–222. DOI: 

  4. World Health Organization. Framework on integrated, people-centred health services. 2016. A69/39, 2018; 1–12. 

  5. Nolte, E and Pitchforth, E. What is the evidence on the economic impacts of integrated care? 2014. 

  6. Conklin, A, Nolte, E and Vrijhoef, H. Approaches to chronic disease management evaluation in use in Europe: a review of current methods and performance measures. International journal of technology assessment in health care, 2013; 29(1): 61–70. DOI: 

  7. Øvretveit, J. Evidence: Does clinical coordination improve quality and save money? 2011; 1(1). The Health Foundation. 

  8. Ouwens, M, Wollersheim, H, Hermens, R, Hulscher, M and Grol, R. Integrated care programmes for chronically ill patients: a review of systematic reviews. International journal for quality in health care, 2005; 17(2): 141–146. DOI: 

  9. Desmedt, M, Vertriest, S, Hellings, J, Bergs, J, Dessers, E, Vankrunkelsven, P, … and Vandijck, D. Economic impact of integrated care models for patients with chronic diseases: a systematic review. Value in Health, 2016; 19(6): 892–902. DOI: 

  10. Davy, C, Bleasel, J, Liu, H, Tchan, M, Ponniah, S and Brown, A. Effectiveness of chronic care models: opportunities for improving healthcare practice and health outcomes: a systematic review. BMC health services research, 2015; 15(1): 194. DOI: 

  11. Siouta, N, Van Beek, K, Van der Eerden, ME, Preston, N, Hasselaar, JG, Hughes, S, … and Radbruch, L. Integrated palliative care in Europe: a qualitative systematic literature review of empirically-tested models in cancer and chronic disease. BMC palliative care, 2016; 15(1): 56. DOI: 

  12. Boland, MR, Tsiachristas, A, Kruis, AL, Chavannes, NH and Rutten-van Mölken, MP. The health economic impact of disease management programs for COPD: a systematic literature review and meta-analysis. BMC pulmonary medicine, 2013; 13(1): 40. DOI: 

  13. de Bruin, SR, Heijink, R, Lemmens, LC, Struijs, JN and Baan, CA. Impact of disease management programs on healthcare expenditures for patients with diabetes, depression, heart failure or chronic obstructive pulmonary disease: a systematic review of the literature. Health policy, 2011; 101(2): 105–121. DOI: 

  14. Damery, S, Flanagan, S and Combes, G. Does integrated care reduce hospital activity for patients with chronic diseases? An umbrella review of systematic reviews. BMJ open, 2016; 6(11): e011952. DOI: 

  15. Tsiachristas, A, Stein, KV, Evers, S and Rutten-van Mölken, M. Performing economic evaluation of integrated care: highway to hell or stairway to heaven? International journal of integrated care, 2016; 16(4). DOI: 

  16. Drummond, MF and Jefferson, TO. Guidelines for authors and peer reviewers of economic submissions to the BMJ. Bmj, 1996; 313(7052): 275–283. DOI: 

  17. Husereau, D, Drummond, M, Petrou, S, Carswell, C, Moher, D, Greenberg, D, … and ISPOR Health Economic Evaluation Publication Guidelines-CHEERS Good Reporting Practices Task Force. Consolidated health economic evaluation reporting standards (CHEERS)—explanation and elaboration: a report of the ISPOR health economic evaluation publication guidelines good reporting practices task force. Value in Health, 2013; 16(2): 231–250. DOI: 

  18. Wong, WP, Feng, J, Pwee, KH and Lim, J. A systematic review of economic evaluations of cardiac rehabilitation. BMC health services research, 2012; 12(1): 243. DOI: 

  19. Drummond, MF, Sculpher, MJ, Claxton, K, Stoddart, GL and Torrance, GW. Methods for the economic evaluation of health care programmes; 2015. Oxford university press. 

  20. Drummond, M, Brandt, A, Luce, B and Rovira, J. Standardizing methodologies for economic evaluation in health care: practice, problems, and potential. International journal of technology assessment in health care, 1993; 9(1): 26–36. DOI: 

  21. Drummond, M, Barbieri, M, Cook, J, Glick, HA, Lis, J, Malik, F, … and Severens, J. Transferability of economic evaluations across jurisdictions: ISPOR Good Research Practices Task Force report. Value in health, 2009; 12(4): 409–418. DOI: 

  22. Dávalos, ME, French, MT, Burdick, AE and Simmons, SC. Economic evaluation of telemedicine: review of the literature and research guidelines for benefit–cost analysis. Telemedicine and e-Health, 2009; 15(10): 933–948. DOI: 

  23. Ramsey, S, Willke, R, Briggs, A, Brown, R, Buxton, M, Chawla, A, … and Reed, S. Good research practices for cost-effectiveness analysis alongside clinical trials: the ISPOR RCT-CEA task force report. Value in health, 2005; 8(5): 521–533. DOI: 

  24. Drummond, M, Manca, A and Sculpher, M. Increasing the generalizability of economic evaluations: recommendations for the design, analysis, and reporting of studies. International journal of technology assessment in health care, 2005; 21(2): 165–171. DOI: 

  25. International Care Foundation. “Integrated Care search,” 2018. [Online]. Available: 

  26. Kodner, DL and Spreeuwenberg, C. Integrated care: meaning, logic, applications, and implications–a discussion paper. International journal of integrated care, 2002; 2. DOI: 

  27. Huang, X, Lin, J and Demner-Fushman, D. Evaluation of PICO as a knowledge representation for clinical questions. In AMIA annual symposium proceedings, 2006; 359. American Medical Informatics Association. 

  28. Steuten, LMG, Vrijhoef, HJM, Van Merode, GG, Severens, JL and Spreeuwenberg, C. The Health Technology Assessment-disease management instrument reliably measured methodologic quality of health technology assessments of disease management. Journal of clinical epidemiology, 2004; 57(9): 881–888. DOI: 

  29. Thomas, J and Harden, A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC medical research methodology, 2008; 8(1): 45. DOI: 

  30. Bradley, EH, Curry, LA and Devers, KJ. Qualitative data analysis for health services research: developing taxonomy, themes, and theory. Health services research, 2007; 42(4): 1758–1772. DOI: 

  31. van Leeuwen, KM, Bosmans, JE, Jansen, AP, Hoogendijk, EO, Muntinga, ME, van Hout, HP, … and van Tulder, MW. Cost-effectiveness of a chronic care model for frail older adults in primary care: economic evaluation alongside a stepped-wedge cluster-randomized trial. Journal of the American Geriatrics Society, 2015; 63(12): 2494–2504. DOI: 

  32. Wiley-Exley, E, Domino, ME, Maxwell, J and Levkoff, SE. Cost-effectiveness of integrated care for elderly depressed patients in the PRISM-E study. Journal of Mental Health Policy and Economics; 2009. 

  33. Karow, A, Reimer, J, König, HH, Heider, D, Bock, T, Huber, C, … and Schulz, H. Cost-effectiveness of 12-month therapeutic assertive community treatment as part of integrated care versus standard care in patients with schizophrenia treated with quetiapine immediate release (ACCESS trial). The Journal of clinical psychiatry, 2012; 73(3): e402–8. DOI: 

  34. Tanajewski, L, Franklin, M, Gkountouras, G, Berdunov, V, Edmans, J, Conroy, S, … and Elliott, RA. Cost-effectiveness of a specialist geriatric medical intervention for frail older people discharged from acute medical units: economic evaluation in a two-Centre randomised controlled trial (AMIGOS). PloS one, 2015; 10(5): e0121340. DOI: 

  35. Lambeek, LC, Bosmans, JE, Van Royen, BJ, Van Tulder, MW, Van Mechelen, W and Anema, JR. Effect of integrated care for sick listed patients with chronic low back pain: economic evaluation alongside a randomised controlled trial. Bmj, 2010; 341: c6414. DOI: 

  36. Bertelsen, JB, Dehbarez, NT, Refsgaard, J, Kanstrup, H, Johnsen, SP, Qvist, I, … and Christensen, KL. Shared care versus hospital-based cardiac rehabilitation: a cost-utility analysis based on a randomised controlled trial. Open heart, 2018; 5(1): e000584. 

  37. Everink, IH, van Haastregt, JC, Evers, SM, Kempen, GI and Schols, JM. An economic evaluation of an integrated care pathway in geriatric rehabilitation for older patients with complex health problems. PloS one, 2018; 13(2): e0191851. DOI: 

  38. Wong, FKY, So, C, Ng, AYM, Lam, PT, Ng, JSC, Ng, NHY, … and Sham, MMK. Cost-effectiveness of a transitional home-based palliative care program for patients with end-stage heart failure. Palliative medicine, 2018; 32(2): 476–484. DOI: 

  39. Uittenbroek, RJ, van Asselt, AD, Spoorenberg, SL, Kremer, HP, Wynia, K and Reijneveld, SA. Integrated and Person-Centered Care for Community-Living Older Adults: A Cost-Effectiveness Study. Health services research, 2018; 53(5): 3471–3494. DOI: 

  40. Tsiachristas, A, Burgers, L and Rutten-van Mölken, MP. Cost-effectiveness of disease management programs for cardiovascular risk and COPD in the Netherlands. Value in Health, 2015; 18(8): 977–986. DOI: 

  41. Lanzeta, I, Mar, J and Arrospide, A. Cost-utility analysis of an integrated care model for multimorbid patients based on a clinical trial. Gaceta sanitaria, 2016; 30(5): 352–358. DOI: 

  42. Celano, CM, Healy, B, Suarez, L, Levy, DE, Mastromauro, C, Januzzi, JL and Huffman, JC. Cost-effectiveness of a collaborative care depression and anxiety treatment program in patients with acute cardiac illness. Value in Health, 2016; 19(2): 185–191. DOI: 

  43. Goorden, M, Vlasveld, MC, Anema, JR, van Mechelen, W, Beekman, AT, Hoedeman, R, … and Hakkaart-van Roijen, L. Cost-utility analysis of a collaborative care intervention for major depressive disorder in an occupational healthcare setting. Journal of occupational rehabilitation, 2014; 24(3): 555–562. DOI: 

  44. Donohue, JM, Belnap, BH, Men, A, He, F, Roberts, MS, Schulberg, HC, … and Rollman, BL. Twelve-month cost-effectiveness of telephone-delivered collaborative care for treating depression following CABG surgery: a randomized controlled trial. General hospital psychiatry, 2014; 36(5): 453–459. DOI: 

  45. Vroomen, JLM, Boorsma, M, Bosmans, JE, Frijters, DH, Nijpels, G and Van Hout, HP. Is it time for a change? A cost-effectiveness analysis comparing a multidisciplinary integrated care model for residential homes to usual care. PloS one, 2012; 7(5): e37444. DOI: 

  46. Looman, WM, Huijsman, R, Bouwmans-Frijters, CA, Stolk, EA and Fabbricotti, IN. Cost-effectiveness of the ‘Walcheren Integrated Care Model’ intervention for community-dwelling frail elderly. Family practice, 2016; 33(2): 154–160. DOI: 

  47. Rosenheck, R, Leslie, D, Sint, K, Lin, H, Robinson, DG, Schooler, NR, … and Correll, CU. Cost-effectiveness of comprehensive, integrated care for first episode psychosis in the NIMH RAISE Early Treatment Program. Schizophrenia bulletin, 2016; 42(4): 896–906. DOI: 

  48. Sahlen, KG, Boman, K and Brännström, M. A cost-effectiveness study of person-centered integrated heart failure and palliative home care: based on a randomized controlled trial. Palliative medicine, 2016; 30(3): 296–302. DOI: 

  49. Steuten, L, Palmer, S, Vrijhoef, B, Van Merode, F, Spreeuwenberg, C and Severens, H. Cost-utility of a disease management program for patients with asthma. International journal of technology assessment in health care, 2007; 23(2): 184–191. DOI: 

  50. Zulman, DM, Chee, CP, Ezeji-Okoye, SC, Shaw, JG, Holmes, TH, Kahn, JS, … and Asch, SM. Effect of an intensive outpatient program to augment primary care for high-need Veterans Affairs patients: a randomized clinical trial. JAMA internal medicine, 2017; 177(2): 166–175. DOI: 

  51. Weaver, MR, Conover, CJ, Proescholdbell, RJ, Arno, PS, Ang, A, Uldall, KK and Ettner, SL. Cost-effectiveness analysis of integrated care for people with HIV, chronic mental illness and substance abuse disorders. Journal of Mental Health Policy and Economics, 2009; 12(1): 33. 

  52. Schellenberg, JRA, Adam, T, Mshinda, H, Masanja, H, Kabadi, G, Mukasa, O, … and Mgalula, L. Effectiveness and cost of facility-based Integrated Management of Childhood Illness (IMCI) in Tanzania. The Lancet, 2004; 364(9445): 1583–1594. DOI: 

  53. Bird, SR, Noronha, M, Kurowski, W, Orkin, C and Sinnott, H. Integrated care facilitation model reduces use of hospital resources by patients with pediatric asthma. Journal for Healthcare Quality, 2012; 34(3): 25–33. DOI: 

  54. Goltz, L, Degenhardt, G, Maywald, U, Kirch, W and Schindler, C. Evaluation of a program of integrated care to reduce recurrent osteoporotic fractures. Pharmacoepidemiology and drug safety, 2013; 22(3): 263–270. DOI: 

  55. Van Orden, M, Hoffman, T, Haffmans, J, Spinhoven, P and Hoencamp, E. Collaborative mental health care versus care as usual in a primary care setting: a randomized controlled trial. Psychiatric Services, 2009; 60(1): 74–79. DOI: 

  56. Cohen, E, Lacombe-Duncan, A, Spalding, K, MacInnis, J, Nicholas, D, Narayanan, UG, … and Friedman, JN. Integrated complex care coordination for children with medical complexity: a mixed-methods evaluation of tertiary care-community collaboration. BMC health services research, 2012; 12(1): 366. DOI: 

  57. McCall, N, Cromwell, J and Urato, C. Evaluation of Medicare Care Management for High Cost Beneficiaries (CMHCB) Demonstration: Massachusetts General Hospital and Massachusetts General Physicians Organization (MGH). Research Triangle Park, NC: RTI International; 2010. 

  58. Markle-Reid, M, Browne, G, Gafni, A, Roberts, J, Weir, R, Thabane, L, … and Henderson, S. The effects and costs of a multifactorial and interdisciplinary team approach to falls prevention for older home care clients ‘at risk’ for falling: a randomized controlled trial. Canadian Journal on Aging/La Revue canadienne du vieillissement, 2010; 29(1): 139–161. DOI: 

  59. Hébert, R, Tourigny, A and Raîche, M. (eds.) Integration of services for disabled people: research leading to action. Edisem; 2008. 

  60. Pozzilli, C, Brunetti, M, Amicosante, AMV, Gasperini, C, Ristori, G, Palmisano, L and Battaglia, M. Home based management in multiple sclerosis: results of a randomised controlled trial. Journal of Neurology, Neurosurgery & Psychiatry, 2002; 73(3): 250–255. DOI: 

  61. Tzeng, DS, Lian, LC, Chang, CU, Yang, CY, Lee, GT and Pan, P. Healthcare in schizophrenia: effectiveness and progress of a redesigned care network. BMC health services research, 2007; 7(1): 129. DOI: 

  62. Blom, J, den Elzen, W, van Houwelingen, AH, Heijmans, M, Stijnen, T, Van den Hout, W and Gussekloo, J. Effectiveness and cost-effectiveness of a proactive, goal-oriented, integrated care model in general practice for older people. A cluster randomised controlled trial: Integrated Systematic Care for older People—the ISCOPE study. Age and ageing, 2016; 45(1): 30–41. DOI: 

  63. Pimperl, A, Schulte, T, Mühlbacher, A, Rosenmöller, M, Busse, R, Groene, O, … and Hildebrandt, H. Evaluating the impact of an accountable care organization on population health: the quasi-experimental design of the german gesundes kinzigtal. Population health management, 2017; 20(3): 239–248. DOI: 

  64. Olsson, LE, Hansson, E, Ekman, I and Karlsson, J. A cost-effectiveness study of a patient-centred integrated care pathway. Journal of advanced nursing, 2009; 65(8): 1626–1635. DOI: 

  65. Simon, GE, Katon, WJ, VonKorff, M, Unützer, J, Lin, EH, Walker, EA, … and Ludman, E. Cost-effectiveness of a collaborative care program for primary care patients with persistent depression. American Journal of Psychiatry, 2001; 158(10): 1638–1644. DOI: 

  66. Bergmann, JN, Legins, K, Sint, TT, Snidal, S, Amor, YB, McCord, GC and UNICEF Research Group. Outcomes and Cost-Effectiveness of Integrating HIV and Nutrition Service Delivery: Pilots in Malawi and Mozambique. AIDS and Behavior, 2017; 21(3): 703–711. DOI: 

  67. Renaud, A, Basenya, O, De Borman, N, Greindl, I and Meyer-Rath, G. The cost effectiveness of integrated care for people living with HIV including antiretroviral treatment in a primary health care centre in Bujumbura, Burundi. AIDS care, 2009; 21(11): 1388–1394. DOI: 

  68. Weeks, WB, Gottlieb, DJ, Nyweide, DJ, Sutherland, JM, Bynum, J, Casalino, LP, … and Fisher, ES. Higher health care quality and bigger savings found at large multispecialty medical groups. Health Affairs, 2010; 29(5): 991–997. DOI: 

  69. Wise, CG, Bahl, V, Mitchell, R, West, BT and Carli, T. Population-based medical and disease management: an evaluation of cost and quality. Disease management, 2006; 9(1): 45–55. DOI: 

  70. Salmon, RB, Sanderson, MI, Walters, BA, Kennedy, K, Flores, RC and Muney, AM. A collaborative accountable care model in three practices showed promising early results on costs and quality of care. Health Affairs, 2012; 31(11): 2379–2387. DOI: 

  71. Koch, TG, Wendling, BW and Wilson, NE. How vertical integration affects the quantity and cost of care for Medicare beneficiaries. Journal of health economics, 2017; 52: 19–32. DOI: 

  72. Hébert, R, Veil, A, Raîche, M, Dubois, MF, Dubuc, N and Tousignant, M. Evaluation of the implementation of PRISMA, a coordination-type integrated service delivery system for frail older people in Québec. Journal of integrated care, 2008; 16(6): 4–14. DOI: 

  73. Paulden, M. Discounting and the evaluation of health care programs. Canadian Agency for Drugs and Technologies in Health; 2016. 

  74. Althaus, F, Paroz, S, Hugli, O, Ghali, WA, Daeppen, JB, Peytremann-Bridevaux, I and Bodenmann, P. Effectiveness of interventions targeting frequent users of emergency departments: a systematic review. Annals of emergency medicine, 2011; 58(1): 41–52. DOI: 

  75. Garrison, LP, Jr., Mansley, EC, Abbott, TA, III, Bresnahan, BW, Hay, JW and Smeeding, J. Good Research Practices for Measuring Drug Costs in Cost-Effectiveness Analyses: A Societal Perspective: The ISPOR Drug Cost Task Force Report—Part II. Value in Health, 2010; 13(1): 8–13. DOI: 

  76. Allen, N, Liberti, L, Walker, SR and Salek, S. A comparison of reimbursement recommendations by European HTA agencies: is there opportunity for further alignment? Frontiers in pharmacology, 2017; 8: 384. DOI: 

  77. Gilbody, S, Bower, P and Whitty, P. “Costs and consequences of enhanced primary care for depression: Systematic review of randomised economic evaluations,” Br. J. Psychiatry, 2006; 189: 297–308. DOI: 

  78. Wilkinson, JR and Murray, SA. Assessment in primary care: practical issues and possible approaches. Bmj, 1998; 316(7143): 1524–1528. DOI: 

  79. Serxner, S, Baker, K and Gold, D. Guidelines for analysis of economic return from health management programs. American Journal of Health Promotion, 2006; 20(6): 1–18. DOI: 

  80. Torgerson, DJ. Contamination in trials: is cluster randomisation the answer? Bmj, 2001; 322(7282): 355–357. DOI: 

  81. Austin, PC. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate behavioral research, 2011; 46(3): 399–424. DOI: 

  82. Austin, PC and Stuart, EA. Moving towards best practice when using inverse probability of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects in observational studies. Statistics in medicine, 2015; 34(28): 3661–3679. DOI: 

  83. Stuart, EA, Duckworth, K, Simmons, J and Barry, CL. “Estimate the Effects of a Policy Change,” 2015; 14(4): 166–182. DOI: 

  84. Knai, C, Nolte, E, Brunn, M, Elissen, A, Conklin, A, Pedersen, JP, … and Fullerton, B. Reported barriers to evaluation in chronic care: experiences in six European countries. Health Policy, 2013; 110(2–3): 220–228. DOI: 

  85. Rowe, AK, De Savigny, D, Lanata, CF and Victora, CG. How can we achieve and maintain high-quality performance of health workers in low-resource settings? The Lancet, 2005; 366(9490): 1026–1035. DOI: 

  86. Goodwin, N. Are networks the answer to achieving integrated care? 2008. DOI: 

  87. Shaw, S, Rosen, R and Rumbold, B. What is integrated care. London: Nuffield Trust, 2011; 7. 

  88. Goodwin, N, Stein, V and Amelung, V. What is integrated care?. In Handbook integrated care 2017; 3–23. Cham: Springer. DOI: 

  89. Eastwood, JG, Kemp, LA and Jalaludin, BB. Realist theory construction for a mixed method multilevel study of neighbourhood context and postnatal depression. SpringerPlus, 2016; 5(1): 1081. DOI: 

  90. Rutten-van Mölken, M, Leijten, F, Hoedemakers, M, Tsiachristas, A, Verbeek, N, Karimi, M, … and Czypionka, T. Strengthening the evidence-base of integrated care for people with multi-morbidity in Europe using Multi-Criteria Decision Analysis (MCDA). BMC health services research, 2018; 18(1): 576. DOI: