Health systems integration: state of the evidence
International Journal of Integrated Care, 17 June 2009 - ISSN 1568-4156
Research and Theory
Health systems integration: state of the evidence
Gail D. Armitage, BA, MA, Analyst, Health Systems and Workforce Research Unit, Alberta Health Services – Calgary (formerly Calgary Health Region), 10301 Southport Lane SW, Calgary, Alberta, Canada T2W 1S7
Esther Suter, PhD, MSW, Research and Evaluation Consultant, Health Systems and Workforce Research Unit, Alberta Health Services – Calgary (formerly Calgary Health Region), 10301 Southport Lane SW, Calgary, Alberta, Canada T2W 1S7
Nelly D. Oelke, RN, BScN, MN, PhD (Candidate), Research and Evaluation Consultant, Health Systems and Workforce Research Unit, Alberta Health Services – Calgary (formerly Calgary Health Region), 10301 Southport Lane SW, Calgary, Alberta, Canada T2W 1S7
Carol E. Adair, BA, MSc, PhD, Associate Professor, Department of Psychiatry and Community Health Sciences, University of Calgary, 3330 Hospital Drive NW, Calgary, Alberta, Canada T2N 4N1
Correspondence to: Gail D. Armitage, BA, MA, Analyst, Health Systems and Workforce Research Unit, Alberta Health Services – Calgary, 10301 Southport Lane SW, Calgary, Alberta, Canada T2W 1S7. Phone: +403-943-0783, Fax: +403-943-2875, E-mail:

Introduction: Integrated health systems are considered a solution to the challenge of maintaining the accessibility and integrity of healthcare in numerous jurisdictions worldwide. However, decision makers in a Canadian health region indicated they were challenged to find evidence-based information to assist with the planning and implementation of integrated healthcare systems.

Methods: A systematic literature review of peer-reviewed literature from health sciences and business databases, and targeted grey literature sources.

Results: Despite the large number of articles discussing integration, significant gaps in the research literature exist. There was a lack of high quality, empirical studies providing evidence on how health systems can improve service delivery and population health. No universal definition or concept of integration was found and multiple integration models from both the healthcare and business literature were proposed in the literature. The review also revealed a lack of standardized, validated tools that have been systematically used to evaluate integration outcomes. This makes measuring and comparing the impact of integration on system, provider and patient level challenging.

Discussion and conclusion: Healthcare is likely too complex for a one-size-fits-all integration solution. It is important for decision makers and planners to choose a set of complementary models, structures and processes to create an integrated health system that fits the needs of the population across the continuum of care. However, in order to have evidence available, decision makers and planners should include evaluation for accountability purposes and to ensure a better understanding of the effectiveness and impact of health systems integration.

health systems integration; systematic literature review; evidence-based policy decisions

Provincial and federal government policy reports have highlighted the need to address Canada’s increasing healthcare service costs and utilization [13]. Integrated health systems have been promoted as a means to build a more effective and efficient healthcare system that takes a patient centred focus and better meets the needs of the populations served [48]. Understanding what is being integrated and for what purpose is necessary in order to identify and implement appropriate models, processes, strategies and structures within the context of population needs.

Evidence-based decision-making as the ‘foundation for an effective and efficient health system’ [9] has been endorsed by a number of Canadian Health Organizations including Health Canada [9] and the Canadian Health Services Research Foundation [10]. Despite a proliferation of literature on integration, decision makers are challenged to find comprehensive, easily accessible, evidence-based information.

Systematic literature reviews have a role to play in providing planners and decision makers with a synthesis of current evidence-based knowledge [9, 1115] to assist with the planning and implementation of health systems integration. Typically, systematic literature reviews involve several steps intended to ensure that (a) the research is relevant to the intended users, (b) the best quality studies are included in the synthesis, (c) diverse research is incorporated by including both qualitative and quantitative findings and often both peer-reviewed and grey literature, and (d) analysis, interpretation and synthesis of the study results is valid and meaningful [1620]. The process is both systematic and documented such that, if repeated, similar results and conclusions would be found, and thus reflects the replicability principle of the scientific method more broadly. The benefits of systematic literature reviews for decision makers and planners include: (a) less reliance on a single research study and the opportunity to view possibly divergent studies together, facilitating a comparison of the findings and providing a synthesis of the challenges and advantages of models, processes, or strategies from a variety of perspectives [14, 21, 22], and (b) time-savings since the underpinning of systematic literature reviews is the rigorous evaluation and synthesis of the findings reported in the literature making for easier access to evidence-based information [14, 21].

The authors conducted a systematic literature review and synthesis of peer-reviewed health sciences and business literature and select grey literature to fill a decision maker identified gap about healthcare integration at the system level. The current manuscript discusses the concepts and definitions of integration, and the reported models, measurement tools, indicators and outcomes of integration that were found. The literature review also identified 10 universal principles common to successfully integrated healthcare systems. These findings are reported elsewhere [23]. The full report can be accessed at [24].


The methods of this review were based on recommendations for systematic review for evidence-based clinical practice [25, 26] with adaptations for the review’s broader health systems and policy related questions [e.g. 13, 27, 28], and the overall principle of replicability. Many of the methods for this review were modeled after an earlier systematic literature review undertaken by one of the co-investigators [27].

There were four major components to the review: (a) developing and validating the research questions, (b) searching for, selecting, rating, and summarizing the peer-reviewed health sciences and business literatures, (c) searching for and selecting grey literature, and (d) synthesizing and report writing.

Early involvement of decision makers was instrumental to ensure the relevance of the literature review. Nineteen decision makers in Alberta, Canada identified primary topics of interest; these included the definitions, models and outcomes of integration, and the characteristics of successfully integrated systems. Two focus groups provided direction for the research questions which were then validated by managers and decision makers. These groups comprised senior management, planners, medical leaders, directors, and managers of programs from within Alberta healthcare, senior policy advisors with the provincial department of health, and others. They represented portfolios across the continuum of care including acute care, community, primary care, rural jurisdictions, urban centres, and public health.

The systematic literature review included peer-reviewed health sciences and business databases. Business databases were searched in order to identify recent innovations in the planning and implementation of integrated systems outside healthcare that may be applicable to the healthcare context.

A test of the preliminary search criteria of the peer-reviewed health sciences and business databases yielded 104,252 items and 48,229 items, respectively. The search terms and date ranges were revised in order to reduce yield results. The revised search parameters (Table 1) resulted in 3234 health science abstracts and 1134 business abstracts (after duplicate removal) which were blind rated for relevancy by three researchers. Abstract rating criteria, based on previously developed scales, were applied as follows: Y*=seems extremely relevant to the research questions (i.e. the central focus is integration and high quality empirical evidence is presented), Y=informs the research questions (i.e. similar to Y* but evidence of lesser quality, or a systematic review, or a concept without presentation of data), M=might inform the research questions (i.e. the topic is integration, but the focus is on some very narrow or tangential aspect), and n=does not inform the research questions (i.e. the abstract is not specific to health services integration or organizational integration, even when these phrases are used). Numeric values were assigned (Y*=3, Y=2, M=1, n=0) and summary scores calculated.

Articles were obtained for abstracts with summary scores ≥5 out of 9 (health sciences n=266; business n=60). These articles were reviewed to determine appropriateness and included if they would, or were likely to, inform the research questions, resulting in 190 health science articles and 29 business articles. Each of these 219 articles were rated by two researchers for quality, using criteria for empirical or non-empirical studies. The empirical rating score (maximum 15 points; 0=not present, 1=present but low quality, 2=present and mid-range quality, 3=present and high quality) was based on the quality of the literature review, research questions and design, population and sampling, data collection and capture, and analysis and reporting of results. The reviewers’ scores were combined for a total of a maximum of 30 points. The non-empirical articles were scored along a 10-point scale, ranging from 1 or 2=best not to include (e.g. poor logic, narrow frame of reference, few interesting ideas) to 10=critical to include (e.g. directly on topic, evidence of critical thought, strong conceptualization). A review of the bibliographies of these articles provided an additional 36 articles which provided historical context. A second search of the health sciences databases yielded 22 relevant articles from the period April 2006 through January 2007 which were included in the review.

A search of the grey literature was undertaken to capture non peer-reviewed literature relevant to the review. Sources included conference proceedings, and select government, health associations’ and agencies’ websites. A Google™ search was also conducted using the same search terms as were used for the peer-reviewed health systems literature search. A process similar to that used for the peer-reviewed literature determined inclusion of articles. Approximately 142 documents were identified, 120 were judged to be relevant to the review and the document material was integrated into the draft report. The majority of these documents were obtained from government and health association websites (e.g. Alberta Heritage Foundation for Medical Research; Australian Division of General Practice; Canadian Health Services Research Foundation; U.K. Integrated Care Network (now DH Care Networks); US Department of Health and Human Services, Health Resources and Services Administration, Health Disparities Collaborative).

The complete first draft of the report, which was a compilation of sections written by team members, was reviewed by all the principal investigators and the final draft was circulated to stakeholders and additional internal and external reviewers for input. Additional revisions, based upon their comments, were made to produce the final report [24].

Jurisdictions and quality of the review documents

The health sciences literature yielded documents from several jurisdictions including the US (46.5%), Canada (18.5%), UK (10.0%), Australia/New Zealand (7.3%), and Europe (5.8%). Similarly, the business literature reported the most findings from the US (43.5%), followed by 13% from each of the UK, Europe and Australia, and 8.7% from Canada.

Less than half of the articles rated as relevant from the health sciences literature (40.7%) and business literature (48.2%) were empirical studies. Of those, the quality rating for the health sciences literature was considerably lower than for the business literature (15.1 vs. 22.8, out of 30, respectively). This was also the case for non-empirical documents (10.1 vs. 13.8, out of 20, health sciences and business literature, respectively).

Concepts and definitions

This literature review did not find a universal definition or concept of integration. More than 70 terms and phrases related to integration were retrieved during this review yielding about 175 definitions and concepts. While many of the terms and phrases entailed only one definition, multiple definitions were found for others. For example, the phrase integrated care was explicitly defined by several authors [e.g. 2936]. While most of these definitions were similar and referred to continuity of care within the healthcare system, one expanded that scope to include social services (e.g. housing and meals). Other differences amongst these definitions included the need to be cost-effective, applicability to patients with complex health needs, or a focus on population based care. The review found numerous concepts of integration, e.g. virtual, vertical, horizontal, functional, clinical, and physician. Additionally, integration could be taken to mean integration at the system level or the coordination of services or programs for a particular population. An example of multiple phrases for a single concept is the mechanism for delivering integrated care which yielded a multitude of terms or phrases, e.g. integrated delivery networks [37], integrated health networks [3840], and integrated health delivery systems [41].


Our literature review revealed a number of health systems integration models but no model was predominant. We categorized the models into three major groups: system level, program/service level, and progressive or sequential models.

The system level models varied considerably but most models focused on aspects of organizational change. In Miller’s [42] causal model of organizational performance and change, leadership plays a central role in positively or negatively impacting the interactions amongst individual performance, organizational performance and organizational culture. The model used by the US’ Veterans’ Affairs Upper Midwest Health Care Network, a publicly funded integrated system, links structures and processes with system level integration and performance [43]. The relational systems change model [44] is premised on the theory that supportive organizations that manage change and encourage and facilitate people to develop, grow, and change within the context of their relationships with others can effect system change.

Program or service level models aim to improve patient outcomes with better coordination of services by focusing on case management [4548], the co-location of services and information [46, 48], implementation of healthcare teams [46], enhanced role of the primary care physicians [49], or the use of a population health approach [50] to facilitate health systems integration.

A common attribute of progressive or sequential models was that health system integration was not a final destination but rather a means of achieving improved healthcare performance while adding value for the system, program, community, patients and providers [51]. Each of the sequential models proposed a number of stages or progressions from less coordinated care to a fully integrated healthcare system [5, 52, 56].

The business literature yielded a number of models we deemed relevant and applicable to the healthcare context. Most of these models were aimed at ensuring the right product went to the right customer at the right time through the right channels [57]. For example, a number of supply chain management and organizational integration models adopted a customer focus [5860]; other models promoted information and knowledge sharing [6164] and cooperation amongst departments to enhance customer service and integration [57, 60, 65, 66].

While we were unable to identify a unified integration model, the models described above highlight some of the essential components of successful integration, which are reported in another article [23], such as using a patient focus, offering the right services to the right clients across the continuum of care, strong leadership, accountability through performance measurement, information sharing across the system, focus on primary care, and healthcare teams. A combination of these factors within a population needs context is likely to facilitate successful integration [23].

Measuring health system integration

Measuring health system integration involves measuring and evaluating both whether the process of integration was implemented as intended showing that an integrated system has been achieved, and the impact of integration on the various components including patient, provider, organization and system, that is, how well the integrated system performed. Overall, our literature search revealed a limited number of clearly and fully described measurement tools and indicators assessing the implementation of integration. As well, actual empirical research on outcomes and impact of integrated health systems was scarce.

Measurement tools

Of the relevant measurement tools identified by this review, the one most frequently used was the balanced scorecard. The balanced scorecard is appropriate for evaluating both the implementation of integration and the impact of integration. It was developed by Kaplan and Norton [67, 68] in response to organizational performance measurements that were based on financial performance alone. The degree of integration implementation can be determined by a survey administered to organization managers [51, 69, 70]. To determine the impact of integration, the organization must consider system-wide relationships, choose the components appropriate for measurement and identify the key indicators within each of those components to ascertain integration outcomes [51, 69]. The balanced scorecard can also be used to track progress by identifying goals and the strategies required for achieving those goals [69]. Several authors proposed various applications of the balanced scorecard. In Canada, the Sisters of Charity of Ottawa Health Services used a balanced scorecard approach to align strategy and performance in long-term care [71], Pink et al. [72] discuss the creation of a balanced scorecard for Ontario’s hospital systems, and the London Health Sciences Centre developed a balanced scorecard to evaluate an integrated dialysis delivery network [73]. In the US both the Mayo Clinic [74] and Veteran’s Affairs Upper Midwest Health Care Network [43] have implemented a balanced scorecard tool. Duke University Health System applied the balanced scorecard to its birthing centre [75] and Bilkhu-Thompson [76] reports on the implementation of a balanced scorecard by an emergency medicine service line. In the UK, the St. Andrew’s Group of Hospitals [77] used the balanced scorecard approach to determine the effectiveness of their health services integration efforts.

The second tool found in the literature was the clinical microsystem assessment tool, which was developed through the systematic analysis of 20 high performing clinical microsystems in North America [78, 79]. Eight characteristics, shared across the 20 microsystems, were consistently related to high success rates of high quality and cost effective care delivery. These characteristics were used to create this self-assessment tool which allows an organization to compare its characteristics to those considered key to successful integration. The survey questionnaire was tested for content and face validity [79]. The small number of items (10) and free access to the survey questionnaire make this an easy and quick tool for evaluation. The Calgary Health Region has used this tool to measure integration of a new community health centre [80].

The third tool of interest was the scale of functional integration [81], which can be used to analyze intra-organizational, inter-organizational, horizontal, and vertical integration. The scale was tested in a Swedish Local Health Care Network. The authors were able to establish the tool’s validity and reliability. The scale is comprised of a continuum from full segregation to full integration within several categories such as patient referrals and pooled resources. Different professional groups rank their perception of their unit’s integration with other units. These rankings are then compared with the optimum rank as determined by each unit.

Indicators of integration

A number of indicators were proposed to measure the extent to which an integrated health system has been achieved. Some authors focused on indicators for functional, clinical and physician integration [51, 69, 82]. Others used network effectiveness in delivering services [83] and the level of integration amongst different organizations within the network [84] as indicators. Leggat and Leatt [41] developed indicators derived from a combination of structural, process, and outcome measures in order to capture information on both the implementation of integration and the performance of the integrated healthcare system.

Outcomes of integration

The literature was reviewed for evidence of effectiveness and outcomes of integrated health systems. Very few studies reported on the impact of integration and tended to focus on perceived benefits rather than empirically derived outcomes. At the system level, studies reported conflicting results. In some cases, financial performance was better [53, 85] or cost per patient visit was reduced [86] while others found no improvement in financial performance [82]. Denver Health, a well-established integrated health system, reported a reduction in emergency room visits and in length of stay in hospital [87]. Three UK community health care trusts reported flatter organizational structures, that is, fewer management tiers [86]. Provider outcomes of the PROCARE project were both positive and negative. Staff reported greater job satisfaction, a blending of professional cultures into one shared culture, and increased cooperation, teamwork and communication with other agencies which they believed benefited patients [88]. However, frontline staff felt challenged working alongside other healthcare professionals because of different medical and social patient-care philosophies [88]. Hurst et al. [86] report that the new skills and knowledge needed to work in an integrated healthcare team resulted in workload problems and challenges retaining staff. With fewer patients presenting at hospital or staying a shorter time, there was greater dependency on community services with the resulting increase in workload [86]. Table 2 provides additional details related to the studies that reported outcomes of integration.


This literature review was conducted to provide decision makers access to succinct, comprehensive, good quality, evidence-based information to assist with the planning, implementation and evaluation of integrating health systems.

The initial literature search yielded an extraordinary number of articles and reports on integration. While about half of the documents relevant to the review were empirical studies, the overall quality of the literature, in particular from the health sciences literature, was limited (average score of the health sciences literature was 15.1 out of 30, the business literature documents scored 22.8 out of 30). Perceived impacts of integration constituted the majority of the literature reporting outcomes. The scarcity of evidence-based information is problematic as high expectations have been attached to integrated healthcare systems. Furthermore, we found numerous terms related to integration, many of which had several definitions. While this is perhaps not surprising given the diversity of disciplines and fields reporting on integration, this diversity of terminology is overwhelming [89]. It may also indicate a lack of understanding or clarity of the concept of integration [35, 90, 91] and what the benefits are. This further hampers the planning and implementation of integrated health systems and the measurement of integration outcomes.

Planners and decision makers expressed interest in learning about models, structures, and processes that would assist them with the planning and implementation of integrated health systems. While there was perhaps an expectation that the systematic literature review would identify a dominant model; that was not the case. There are many contextual factors that affect the delivery of healthcare services including the diversity of populations served, existing policies and political environment, geographical issues, differing philosophies of care, advances in biotechnology, and funding mechanisms [45, 48, 53]. It is likely that healthcare is too complex for a one-size-fits-all solution. However, the models highlight a number of essential characteristics such as patient centeredness, offering services across the continuum of care, strong leadership, accountability through performance measurement, information sharing across the system, focus on primary care, and healthcare teams. These characteristics, if fully implemented, may facilitate successful integration. It is important for decision makers and planners to choose an optimal set of complementary models, structures and processes to implement integration that fits the contextual situation and the needs of the population across the continuum of care.

Given the increasing demand for accountability, measuring the implementation and impact of integration on system, provider and patient levels is essential. Several instruments have been used throughout the years; however, there is a lack of standardized, validated tools that have been systematically used to evaluate integration implementation and impact. There are several challenges to the development of standardized tools including complexity of healthcare systems, the number of stakeholders who must reach consensus, understanding what is to be integrated and the anticipated outcomes, and the challenge of attributing effects to causes in complex, multifaceted systems. There is a paucity of strong empirical evidence for the impact of integration and the reported impacts are mixed. Nevertheless, some positive system and program/provider level outcomes were reported.

Strengths and limitations of the review

The strengths of this systematic literature review include the validation of the research questions by intended users of the review, the multidisciplinary knowledge base of the research team and a thorough methodological approach. Inclusion of business literature provided some insights that can be generalized to the healthcare field such as patient centred models. In many ways, the information gathered in the healthcare literature was validated by the findings in the business articles. The focus on system level integration could be considered a limitation of the review as program level studies, which may have provided insights to integration efforts, were not targeted. The integration of services for special or vulnerable populations such as those with chronic diseases, HIV/AIDS, or mental illness and children, youth, the elderly, and Aboriginal peoples may be more advanced and closer study should be undertaken.


This systematic review has highlighted some models, measurement tools and outcomes of integration that may inform the planning and implementation of integrated health systems. The review has also revealed some significant gaps which are particularly prevalent in the area of standardized tools to measure integration outcomes including efficiency and effectiveness at the system, program, provider and client levels over time. This lack in evidence-based information hampers our understanding of how to best integrate health systems within different contexts and for different desired outcomes.

In the absence of a unified model for health system integration there is a need to clearly identify the components that are relevant for an integrated system within a given context. Both the health and business literature have yielded a number of models that promise applicability to various contexts and might serve to inform planners and decision makers.

The lack of clearly demonstrated integration outcomes sends a strong message to planners and decision makers to include evaluation for accountability purposes when planning and implementing integrated health systems. In this way, planners and decision makers can identify strategies and processes that assist with the implementation of integration and those that are standing as barriers to successful integration efforts. This will ensure a better understanding of the effectiveness of health systems integration and system performance.

Integration is an ongoing process which must be developed and implemented within the context of population needs and focused on the goals of improved health outcomes and higher quality of care. In order to move integration forward, decision makers must encourage and support a research agenda which includes the development of:

  • A set of clear standards for monitoring success and failure of integrated health systems

  • Validated measurement tools to measure integration outcomes including cost-effectiveness measures

  • Comprehensive case studies that document processes, principles, and challenges in planning and implementing integrated healthcare systems within different contexts as well as the roles of the different stakeholder groups

  • Comparative analyses of different approaches to integration in terms of ease of implementation and level of success.

It is widely recommended that decision makers and planners use evidence-based information [9, 10, 12, 92, 93] to inform their integration efforts. Literature reviews, that are arrived at through systematic, replicable methods, such as the current review, can assist decision makers and planners with time-saving access to rigorously evaluated, synthesised [9, 1215] evidence-based findings. However, it is also true that decision makers and planners take a broad view of evidence including expert opinion, political judgement and social values [94, 95] as well as the more narrowly defined scientific evidence [92, 95]. Given the gaps identified in the evidence-based information on health system integration, it is suggested that the information from this review is used in conjunction with organizational data and other sources of information to inform the planning, execution and evaluation of integrating divergent components within a healthcare system.


The authors would like to acknowledge the following:

  • Canadian Institutes of Health Research for their financial support,

  • Decision makers and planners who guided the development of the project and the refinement of the research questions,

  • Reviewers for their insightful comments and suggestions on the draft report,

  • Librarians for their assistance with the search strategy, and

  • Research team staff.


Corinne Kyriacou, PhD, Assistant Professor, Department of Health Professions & Family Studies School of Education, Health & Human Services Hofstra University Hempstead, NY, USA

Gudjon Magnusson, Dr., Professor of Public Health, Reykjavik University, Reykjavik, Iceland

One anonymous reviewer

Commission on the Future of Health Care in Canada. Building on values: the future of health care in Canada (Romanow Report). Ottawa ON: Commission on the Future of Health Care in Canada; 2002.
Premier’s Advisory Council on Health. A framework for reform. (Mazankowski Report). Edmonton AB: Alberta Health; 2001.
Saskatchewan Commission on Medicare. Caring for Medicare: sustaining a quality system (Fyke Report). Saskatchewan Commission on Medicare 2001.
Coddington DC, Ackerman FK, Jr, Moore KD. Integrated health care systems: major issues and lessons learned. Healthcare Leadership and Management Report 2001 Jan; 9(1):1-9.
Fleury MJ. Integrated service networks: the Quebec case. Health Services Management Research 2006 Aug; 19(3):153-65.
Leatt P, Pink GH, Guerriere M. Towards a Canadian model of integrated healthcare. Healthcarepapers 2000 Spring; 1(2):13-35.
Powell Davies G. Chapter 1: Integration: an overview. In: Powell Davies G, Betbeder-Matibet L, Nicholls A, Reynolds F, Bonney MA, Traynor V, et al, editors. General practice integration: a literature review. Sydney AU: Centre for General Practice Integration Studies; 1996. p. 3-7.
Ross-van Dorp C. Speech to be given by the State Secretary of Health, Welfare and Sport, Clémence Ross-van Dorp, at the opening of the Integrated Care conference. The Hague: 2004 Nov 18. Available from:
Gainer K, Watanabe M, Evans RG, McDonald M, Maldoff M, Noseworthy TW, et al. Creating a culture of evidence-based decision making. Canada health action: building on the legacy—volume II—synthesis reports and issues papers. Ottawa ON: Health Canada; Undated, Date Modified: 2004 May 28 [cited 2008 Oct 6]. Available from:
Canadian Health Services Research Foundation. Health services research and evidence-based decision-making. Ottawa ON: Canadian Health Services Research Foundation; 2000.
Fox DM. Evidence of evidence-based health policy: the politics of systematic reviews in coverage decisions. Health Affairs 2005 Jan–Feb; 24(1):114-22.
Kiefer L, Frank J, Di Ruggiero E, Dobbin M, Manuel D, Gully PR, et al. Fostering evidence-based decision-making in Canada: examining the need for a Canadian population and public health evidence centre and research network. Canadian Journal of Public Health 2005 May–Jun; 96(3):I1-I19. following 200.
Lavis JN, Posada FB, Haines A, Osei E. Use of research to inform public policymaking. Lancet 2004 Oct 30–Nov 5; 364(9445):1615-21.
Lavis JN, Davies H, Oxman A, Denis J, Golden-Biddle K, Ferlie E. Towards systematic reviews that inform health care management and policy-making. Journal of Health Services Research and Policy 2005 Jul; 10(Suppl 1):35-48.
Lomas J. Using research to inform healthcare managers’ and policy makers’ questions: from summative to interpretive synthesis. Healthcare Policy 2005 Sep; 1(1):55-71.
Atkins S, Lewin S, Smith H, Engel M, Fretheim A, Volmink J. Conducting a meta-ethnography of qualitative literature: lessons learnt. BMC Medical Research Methodology [serial online] 2008 [cited 2008 May 2]; 8:21. Available from:
Bravata DM, McDonald KM, Shojania KG, Sundaram V, Owens DK. Challenges in systematic reviews: synthesis of topics related to the delivery, organization, and financing of health care. Annals of Internal Medicine 2005 Jun 21; 142(12 Pt 2):1056-65. W250-3.
Brereton P, Kitchenham BA, Budgen D, Turner M, Khalil M. Lessons from applying the systematic literature review process within the software engineering domain. The Journal of Systems and Software 2007; 80:571-83.
Dixon-Woods M, Fitzpatrick R. Qualitative research in systematic reviews. Has established a place for itself. British Medical Journal 2001 Oct 6; 323(7316):765-6.
Petticrew M. Systematic reviews from astronomy to zoology: myths and misconceptions. British Medical Journal 2001 Jan 13; 322(7278):98-101.
Lavis JN, Davies HTO, Gruen RL, Walshe K, Farquhar CM. Working within and beyond the Cochrane Collaboration to make systematic reviews more useful to healthcare managers and policy makers. Healthcare Policy 2006 Jan; 1(2):21-33.
Wallace A, Croucher K, Bevan M, Jackson K, O’Malley L, Quilgars D. Evidence for policy making: some reflections on the application of systematic reviews to housing research. Housing Studies 2006; 21(2):297-314.
Suter E, Oelke ND, Adair CE, Armitage GD. Ten key principles for successful health systems integration. Healthcare Quarterly 2009. In press.
Suter E, Oelke ND, Adair CE, Waddell C, Armitage GD, Huebner LA. Health systems integration. Definitions, processes and impact: a research synthesis. HSWRU Current research 2007 Oct [cited 2008 Nov 18]. Available from:
Higgins JPT, Green S. Cochrane handbook for systematic reviews of interventions 4.2.6. Chichester JK: John Wiley & Son, Ltd; 2006 [cited 2006 Dec 28]. Available from:
Khan KS, Biet G, Glanville J, Sowden AJ, Kleijnen J. Undertaking systematic reviews of research on effectiveness: CRD’s guidance for those carrying out or commissioning reviews. York, UK: NHS Centre for Reviews and Dissemination; 2001.
Adair CE, Simpson L, Birdsell JM, Omelchuk K, Casebeer AL, Gardiner HP, et al. Performance measurement systems in health and mental health services: models, practices and effectiveness. A state of the science review. Edmonton AB: Alberta Heritage Foundation for Medical Research; 2003 [cited 2006 Feb 6]. Available from:
Wilczynski NL, Haynes RB, Lavis JN, Ramkissoonsingh R, Arnold-Oatley AE. Optimal search strategies for detecting health services research studies in MEDLINE. Canadian Medical Association Journal 2004 Nov 9; 171(10):1179-85.
Aitken JB, Curtis R. Integrated health care: improving client care while providing opportunities for mental health counselors. Journal of Mental Health Counseling 2004; 26:321-31.
Billings JR. What do we mean by integrated care? A European interpretation. Journal of Integrated Care 2005 Oct; 13(5):13-20.
Blount A. Integrated primary care: organizing the evidence. Families, Systems and Health 2003; 21(2):121-33.
Coster G. An academic perspective on the development of integrated care. Healthcare Review Online [serial online] 1998 Aug 1 [cited 2006 May 30]; 2(10). Available from:
Feek C. Policy and ethics. Healthcare Review Online [serial online] 1998 May 1 [cited 2006 May 30]; 2(7). Available from:
Hardy B, Mur-Veeman I, Steenbergen M, Wistow G. Inter-agency services in England and the Netherlands: a comparative study of integrated care development and delivery. Health Policy 1999 Aug; 48(2):87-105.
Kodner DL. The quest for integrated systems of care for frail older persons. Aging, Clinical and Experimental Research 2002 Aug; 14(4):307-13.
Mur-Veeman I, Hardy B, Steenbergen M, Wistow G. Development of integrated care in England and the Netherlands: managing across public-private boundaries. Health Policy 2003 Sep; 65(3):227-41.
Burns LR, Pauly MV. Integrated delivery networks: a detour on the road to integrated health care?. Health Affairs 2002 Jul–Aug; 21(4):128-43.
Adamson BJ, Crook GF, Girotti M, MacLean NF. The creation of strategic alliances: the experience of the London Health Sciences Centre and southwestern Ontario. Healthcare Management Forum 2000 Summer; 13(2):33-41.
Friedman L, Goes J. Why integrated health networks have failed. Frontiers of Health Services Management 2001 Summer; 17(4):3-28.
Luke RD, Begun JW. Have IHNs failed in healthcare?. Frontiers of Health Services Management 2001 Summer; 17(4):45-50. discussion 51–4.
Leggat SG, Leatt P. A framework for assessing the performance of integrated health delivery systems. Healthcare Management Forum 1997 Spring; 10(1):11-8.
Miller JL. A post-mortem on healthcare integration: an organizational development approach. Healthcare Leadership and Management Report 2000 Sep; 8(9):5-15.
Lukas CV, Meterko M, Lowcock S, Donaldson-Parlier R, Blakely M, Davies M, et al. Monitoring the progress of system integration. Quality Management in Health Care 2002 Winter; 10(2):1-11.
Markoff LS, Finkelstein N, Kammerer N, Kreiner P, Prost CA. Relational systems—Change: implementing a model of change in integrating services for women with substance abuse and mental health disorders and histories of trauma. The Journal of Behavioral Health Services and Research 2005 Apr–Jun; 32(2):227-40.
King G, Meyer K. Service integration and co-ordination: a framework of approaches for the delivery of co-ordinated care to children with disabilities and their families. Child Care, Health and Development 2006 Jul; 32(4):477-92.
O’Connell B, Kristjanson L, Orb A. Models of integrated cancer care: a critique of the literature. Australian Health Review 2000; 23(1):163-78.
Weiss ME. Case management as a tool for clinical integration. Advanced Practice Nursing Quarterly 1998 Summer; 4(1):9-15.
Wulsin LR, Sollner W, Pincus HA. Models of integrated care. Medical Clinics of North America 2006 Jul; 90(4):647-77.
Batterham R, Southern D, Appleby N, Elsworth G, Fabris S, Dunt D, et al. Construction of a GP integration model. Social Science and Medicine 2002 Apr; 54(8):1225-41.
Byrnes JJ. Do integrated healthcare strategies enhance quality?. Integrated Healthcare Report 1998 Jul:6-10.
Gillies RR, Shortell SM, Anderson DA, Mitchell JD, Morgan KL. Conceptualizing and measuring integration: findings from the health systems integration study. Hospital and Health Services Administration 1993 Winter; 38(4):467-89.
Boon H, Verhoef M, O’Hara D, Findlay B. From parallel practice to integrative health care: a conceptual framework. BMC Health Services Research [serial online] 2004 Jul 1 [cited 2006 Nov 28]; 4(1):15. Available from:
Conrad DA, Shortell SM. Integrated health systems: promise and performance. Frontiers of Health Services Management 1996 Fall; 13(1):3-40. iscussion 57–8.
Konrad EL. A multidimensional framework for conceptualizing human services integration initiatives. New Directions for Evaluation 1996; 69:5-19.
Leutz W. Five laws for integrating medical and social services: lessons from the United States and the United Kingdom. Milbank Quarterly 1999; 77(1):77-110. iv–v.
Leutz W. Reflections on integrating medical and social care: five laws revisited. Journal of Integrated Care 2005 Oct; 13(5):3-12.
Chan JO. Toward a unified view of customer relationship management. The Journal of American Academy of Business 2005 Mar:32-8. Available from:
Brandao F, Wilken LS, Dutra FA. Samarco integrated management system. Charlotte NC: American Society for Quality; 2001:726-38.
Dan B, Li L, Zhang X, Guo F, Zhou J. Network-integrated manufacturing system. International Journal of Production Research 2005; 43(12):2631-47.
Fawcett SE, Cooper MB. Process integration for competitive success: benchmarking barriers and bridges. Benchmarking: An International Journal 2001; 8(5):396-412.
Beretta S. Unleashing the integration potential of ERP systems. Business Process Management Journal 2002; 8(3):254-77.
Beretta S. Enabling integration in complex contexts: the role of process based performance measurement systems. Managerial Finance 2004; 30(8):69-91.
Braganza A. Enterprise integration: creating competitive capabilities. Integrated Manufacturing Systems 2002; 13(8):562-72.
Ghoshal S, Gratton L. Integrating the enterprise. MIT Sloan Management Review 2002; 44:31-8.
Childerhouse P, Hermiz R, Mason-Jones R, Popp A, Towill DR. Information flow in automotive supply chains—identifying and learning to overcome barriers to change. Industrial Management and Data Systems 2003; 103(7):491-502.
Stonebraker PW, Afifi R. Toward a contingency theory of supply chains. Management Decision 2004; 42(9):1131-44.
Kaplan RS, Norton DP. The balanced scorecard—measures that drive performance. Harvard Business Review 1992 Jan–Feb; 70:71-9.
Kaplan RS, Norton DP. Putting the balanced scorecard to work. Harvard Business Review 1993 Sep; 71:134-47.
Devers KJ, Shortell SM, Gillies RR, Anderson DA, Mitchell JB, Erickson KM. Implementing organized delivery systems: an integration scorecard. Health Care Management Review 1994 Summer; 19(3):7-20.
Shortell SM, Anderson DA, Gillies RR, Mitchell JB, Morgan KL. The holographic organization. The Healthcare Forum Journal 1993; 36:20-6.
Macdonald M. Using the balanced scorecard to align strategy and performance in long-term care. Healthcare Management Forum 1998 Fall; 11(3):33-8.
Pink GH, McKillop I, Schraa EG, Preyra C, Montgomery C, Baker GR. Creating a balanced scorecard for a hospital system. Journal of Health Care Finance 2001 Spring; 27(3):1-20.
Peters K, Ryan H. An integrated dialysis delivery network in Ontario. CANNT Journal 1999 Winter; 9(1):20-3.
Curtright JW, Stolp-Smith SC, Edell ES. Strategic performance management: development of a performance measurement system at the Mayo Clinic. Journal of Healthcare Management 2000 Jan–Feb; 45(1):58-68.
Jones MLH, Filip SJ, Smith H, Remsburg-Bell E. Strategy management system in perinatal services: the role of a patient resource manager. Lippincott’s Case Management 2002 Jan–Feb; 7(1):27-42.
Bilkhu-Thompson MK. A process evaluation of a health care balanced scorecard. Journal of Health Care Finance 2003 Winter; 30(2):37-64.
Sugarman PA, Watkins J. Balancing the scorecard: key performance indicators in a complex healthcare setting. Clinician in Management 2004; 12:129-32.
Mohr JJ, Batalden PB. Improving safety on the front lines: the role of clinical microsystems. Quality and safety in health care 2002 Mar; 11(1):45-50.
Nelson EC, Batalden PB, Huber TP, Mohr JJ, Godfrey MM, Headrick LA, et al. Microsystems in health care: part 1: learning from high-performing front-line clinical units. The Joint Commission Journal on Quality Improvement 2002 Sep; 28(9):472-93.
Suter E, Hyman M, Oelke N. Measuring key integration outcomes: a case study of a large urban health centre. Health Care Management Review 2007 Jul–Sep; 32(3):226-35.
Ahgren B, Axelsson R. Evaluating integrated health care: a model for measurement. International Journal of Integrated Care [serial online] 2005 Aug 31 [cited 2005 Sep 15]; 5. Available from:
Lee K, Wan TT. Effects of hospitals’ structural clinical integration on efficiency and patient outcome. Health Services Management Research 2002 Nov; 15(4):234-44.
Provan KG, Milward HB. Do networks really work? A framework for evaluating public-sector organizational networks. Public Administration Review 2001 Jul–Aug; 61(4):414-23.
Browne G, Roberts J, Gafni A, Kertyzia J, Loney P. Conceptualizing and validating the human services integration measure. International Journal of Integrated Care [serial online] 2004 May 19 [cited 2006 Sep 11]; 4. Available from:
Wang BB, Wan TT, Clement J, Begun J. Managed care, vertical integration strategies and hospital performance. Health Care Management Science 2001 Sep; 4(3):181-91.
Hurst K, Ford J, Gleeson C. Evaluating self-managed integrated community teams. Journal of Management in Medicine 2002; 16(6):463-83.
Gabow P, Eisert S, Wright R. Denver Health: a model for the integration of a public hospital and community health centers. Annals of Internal Medicine 2003 Jan 21; 138(2):143-49.
Coxon K. Common experiences of staff working in integrated health and social care organisations: a European perspective. Journal of Integrated Care 2005; 13(2):13-21.
Boon H, Verhoef M, O’Hara D, Findlay B, Majid N. Integrative healthcare: arriving at a working definition. Alternative Therapies in Health and Medicine 2004 Sep–Oct; 10(5):48-56. Available from:
Marriott J, Mable AL. Integration: follow your instincts, ignore the politics, and keep your eyes on the ideal model. Canadian Journal of Public Health 1998 Sep–Oct; 89(5):293-6. 314.
Simoens S, Scott A. Integrated primary care organizations: to what extent is integration occurring and why?. Health Services Management Research 2005 Feb; 18(1):25-40.
Clements D. 2004 What counts? Interpreting evidence-based decision-making for management and policy. Report of the 6th Canadian Health Services Research Foundation Annual Invitational Workshop; 2005 March 11; Vancouver BC.
Lomas J. Commentary whose views count in evidence synthesis? And when do they count?. Healthcare Policy 2006 Jan; 1(2):55-7.
Canadian Health Services Research Foundation. Weighing up the evidence. Making evidence-informed guidance accurate, achievable, and acceptable. Ottawa ON: Canadian Health Services Research Foundation; 2006.
Lomas J, Culyer T, McCutcheon C, McAuley L, Saw S. Conceptualizing and combining evidence for health system guidance. Ottawa ON: Canadian Health Services Research Foundation; 2005.
Table 1.

Search parameters conducted April 2006

Database Search terms Limits Yield
Health databases
Medline Delivery of health care, integrated 1998–2006
English; focussed; All subheadings except history and ethics
Delivery of healthcare, integrated OR organizational integration OR integrated health services OR integrated healthcare OR integrated care OR integrated service delivery systems OR care co-ordination OR health services integration 1998–2006
Business databases
ABI inform global CBCA Organizational integration OR integrated delivery systems 2001–2006
Scholarly journals
Business source premier Organizational integration OR integrated delivery systems 2001–2006
Scholarly journals
Table 2.

Outcomes of integration

Author(s) Context What was being measured? How was it being measured? What were the findings?
Conrad and Shortell 1996 [53] Study of nine US organized delivery systems (part of Health Systems Integration Study) Degree of functional integration including financial management Questionnaires with managers Better financial performance compared with competitors
Coxon 2005 [88] Part of the PROCARE study Experiences of frontline staff working in integrated health and social care organisations Questionnaires, interviews and focus groups with staff and managers Improvements in:
        • Job satisfaction
        • Teamwork
        • Communication
        • Inter-agency cooperation
        • Shared culture
        • Different professional cultures
Gabow et al. 2003 [87] Comparative analysis amongst Denver Health and Hospital Authority, urban public hospitals, and urban community health centres Hospital utilization Data from 1997 annual surveys, the Bureau of Primary Health Care 1998 uniform data system, and Denver Health and Hospital Authority • Reduction in non-emergency cases using the emergency room
        • Average length of stay in hospitals is reduced
Hurst et al. 2002 [86] Comparative study of three UK community health care (CHC) trusts (similar to Alberta’s Primary Care Networks) Managerial and clinical practices in the CHCs Mixed methods study (interviews, questionnaires, non-participant observation) of primary healthcare providers Benefits:
        • Reduced cost per patient site visit CHCs
        • Flatter organisational structure
        • Workload and staffing problems
Lee and Wan 2002 [82] Study sample was comprised of 358 US community hospitals Relationship between a hospital’s structural clinical integration and average total cost per discharge Data from the 1997 annual surveys, 1999 area resource file, Dorenfest IHDS+ database Higher level of integration does not immediately improve hospital’s financial performance
Wang et al. 2001 [85] Sample consists of 363 California short-term acute care hospitals Relationship between degree of integration and financial performance Data from annual surveys and disclosure reports, health care financial administration, area resource file Integration has a positive effect on financial success