Monitoring the degree of implementation of an integrated delivery system

Abstract Introduction The aim of the study was to develop a method to measure the implementation of specific components of an Integrated Service Delivery system for the frail elderly. The system includes six mechanisms and tools: (1) coordination of all organizations involved in delivering health and social services, (2) a single entry point, (3) case management, (4) a single assessment tool with a case-mix classification system, (5) an individualized service plan, and (6) a computerized clinical chart. Method Focus groups of researchers, clinicians, managers and policy-makers identified quantitative indicators for each component. The six components were weighted according to their relative importance in order to generate a total score. Data were collected every six months over 30 months to establish the implementation degree in the three experimental areas: Sherbrooke, Granit and Coaticook in the Province of Quebec, Canada. Results After 30 months, coordination is the most developed component in the three experimental areas. Overall, in July 2003, the Integrated Service Delivery system was implemented at the rate of 73%, 71% and 70% in Sherbrooke, Granit and Coaticook, respectively. Discussion This type of quantitative assessment provides data for managers and researchers to monitor the implementation. Moreover, when there is an outcome study, the results of the outcome study can be correlated with the degree of implementation, thus allowing for dose-response analyzes and helping to decrease the “black box” effect.


Introduction
Integrated Service Delivery (ISD) systems have been proposed for improving efficacy and efficiency of health care system, particularly for patients with multiple needs and complex interactions of many professionals and organizations. It is hypothesized that ISD systems would improve continuity of care and the health and satisfaction of clients, together with decreasing the use of costly resources, like hospitals and institutions. While there is some indications of the efficacy of ISD systems for some clientele like frail older people w1x, their real effectiveness at the population level remains to be demonstrated. Although many ISD systems have been well described and compared w2-4x, those experimented so far have not reported much on the implementation of the components of the system, nor on its process and real functioning. The lack of implementation data limits the replication of studies and could explain some negative results about the impact of ISD systems. It also deprives policy makers of critical information for applying ISD in the health care system.
The implementation of ISD systems should be assessed and monitored in order to inform managers and policy-makers of how the project is evolving, to account for the use of resources, identify obstacles quickly and find strategies to foster full implementation. Such a process analysis is also helpful in evaluative research to describe facilitating factors and obstacles that future implementation will have to consider. When an outcome evaluation is performed, a process analysis is used to ensure that the intervention really has been implemented as planned. This assessment is usually qualitative, but it is also informative to generate some quantitative ratings, especially when an economic analysis is done. Ratings could then help policy-makers to evaluate the degree of implementation and extrapolate the effort and resources needed for full implementation. In an outcome evaluation study, such ratings can be used to measure the ''dosage'' of the intervention and carry out dose-response analyses. The objective of this study was to develop a methodology to rate the degree of implementation and to monitor the implementation of an ISD system, the PRISMA model in three different areas of the Eastern Township in the Province of Quebec, Canada (the PRISMA-Estrie project).

The PRISMA model
According to Leutz, there are three levels of integration in health care: (1) linkage; (2) coordination; and (3) full integration w5x. ISD refers to systems targeting either coordination or full integration. In full integration ISD systems, the integrated organization is responsible for all services, either under one structure or by contracting some services with other organizations. Many examples of this level of ISD programmes have been developed. In the United States, the California On Lok project w6x gave rise to the PACE (Program of All inclusive Care for the Elderly) projects w7x. In Canada, the CHOICE (Comprehensive Home Option of Integrated Care for the Elderly) project in Edmonton is an adaptation of the PACE projects w4x. These programmes are built around Day Centres where the members of the multidisciplinary team who evaluate and treat the clients are based. Clients are selected according to relatively strict inclusion (degree of disability compatible with admission to a nursing home) and exclusion (e.g. behavioural problems) criteria. These systems usually function in parallel with the socio-health structures in place. Services are delivered by structures operated by the system or by external structures linked through contracts (hospitals, specialized medical care, long-term care institutions). The Social HMO in the United States w8x and the SIPA (''Systé me de services inté gré s pour personnes âgées en perte d'autonomie'') project in Montreal are also integrated services but do not include a Day Centre w3x. However, home care services are provided by personnel hired by or under contract with the organization. All these fully integrated models are nested in the usual health and social services in a particular area but are run in parallel to them. They do not involve significant changes to the structure or processes of existing services, except for the negotiation of protocols for referring clients to ISD and the provision of some services not covered by ISD. Capitation budgeting is usually a key component of these programmes.
Although there were many attempts to design and implement full integration models, very few experiments of coordination models were implemented or tested. The PRISMA (Program of Research to Integrate Services for the Maintenance of Autonomy) model is a new model of integrated care based on coordination w9x. As opposed to full integration systems, this model includes all the public, private or voluntary health and social service organizations involved in caring for older people in a given area. Every organization keeps its own structure but agrees to participate within an ''umbrella'' system and to adapt its operations and resources to the agreed requirements and processes. At this level, the ISD system is not only nested in the health care and social services system (like the full integration models) but is also embedded within it.
The PRISMA model includes six components: (1) coordination between decision-makers and managers at the regional and local level, (2) a single entry point, (3) a case management process, (4) an individualized service plan, (5) a single assessment instrument coupled with a management system based, and (6) a computerized clinical chart. Coordination between institutions is at the core of the PRISMA model. Coordination must be established at every level of the organizations. First, at the strategic level (governance), by creating a Joint Governing Board (''Table de concertation'') of all health care and social services organizations and community agencies where the decision-makers agree on the policies and orientations and what resources to allocate to the integrated system. Second, at the tactical level (management), a Service Coordination Committee, mandated by the Board and comprising public and community service representatives together with older people, monitors the service coordination mechanism and facilitates adaptation of the service continuum. Finally, at the operational level (clinical), a multidisciplinary team of practitioners surrounding the case manager evaluates clients' needs and delivers the required care.
The single entry point is the mechanism for accessing the services of all the health care institutions and community organizations in the area for the frail senior with complex needs. It is a unique gate which older people, family caregivers and professionals can access by telephone or written referral. A link is established with the Health Info Line available to the general population in Quebec seven days a week, 24 hours a day. Clients are referred to the ISD system after a brief needs assessment (triage) to ensure they meet the eligibility criteria for the integrated system. Otherwise, they are referred to the relevant service. ISD eligible clients are then referred to a case man-ager. The case manager is responsible for doing a thorough evaluation of the client's needs, planning the required services, arranging to admit the client to these services, organizing and coordinating support, directing the multidisciplinary team of practitioners involved in the case, and monitoring and re-evaluating the client. The case manager is legitimized to work in all institutions or services. The individualized service plan results from the overall assessment of the client and summarizes the prescribed services and target objectives. It is led by the case manager and established at a meeting of the multidisciplinary team including all the main practitioners involved in caring for the older person.
The single assessment instrument allows for evaluating the needs of clients in all organizations and by all the professionals working either in home care organizations or in hospitals and institutions. The instrument implemented in the PRISMA model is the SMAF (Systeme de mesure de l'autonomie fonctionnelle-Functional Autonomy Measurement System), a 29item scale developed according to the WHO classification of disabilities w10x. It measures functional ability in five areas: activities of daily living (ADL) w7 itemsx, mobility w6 itemsx, communication w3 itemsx, mental functions w5 itemsx and instrumental activities of daily living (IADL) w8 itemsx. For each item, the disability is scored on a 5-point scale: 0 (independent), -0.5 (with difficulty), -1 (needs supervision), -2 (needs help), -3 (dependent). The resources available to compensate for the disability are also evaluated and a handicap score is deducted. A case-mix classification system based on the SMAF has also been developed w11x. Fourteen Iso-SMAF profiles were generated using cluster analysis techniques in order to define groups that are homogeneous in regard to their profiles, but heterogeneous in other respects. These profiles are used to establish the admission criteria to the different services and to calculate the required budget of the organizations, given the autonomy of the clientele served w12x. Finally, the PRISMA model includes a computerized clinical chart for facilitating communications between organizations and professionals. The SIGG (''Systè me d'information gé rontogé riatrique'') has been developed and implemented in a pilot project in Victoriaville (Quebec, Canada). This shareable clinical chart uses the Quebec Ministry of Health and Social Services Internet network and was developed from a Lotus Notes platform.

The PRISMA-Estrie project
The study reported in this paper is part of a larger project assessing the implementation and impact of the PRISMA model. After piloting the model in a different area w13x, we extended the PRISMA model to three areas in another region (Eastern Townships of Quebec) that present different environments: -Sherbrooke: an urban area (population 145,000; 13%)65) with 3 university establishments (a tertiary care hospital, a geriatric institute and a primary care agency), two large public nursing homes (745 beds), several private and public residential facilities, a public home care agency and some voluntary agencies providing complementary services in home care; -Granit: a rural area (population 22,000; 15%)65) with one merged public establishment that includes a local hospital, a primary care program (including home care), a nursing home (106 beds), public and private residential facilities, and some voluntary agencies providing complementary services in home care; -Coaticook: a rural area (population 16,000; 14% )65) with one merged establishment that includes a primary care program, a nursing home (88 beds), an emergency service but no hospital beds, mostly private residential facilities and some voluntary agencies providing complementary services in home care.
The implementation evaluation focuses on the process of implementing the mechanisms and tools and how they function. One of the objectives is to explain the variations observed between the different implementation settings using a case study approach (multiple case study design) developed by Yin w14x. The questions that are documented try to define the extent to which the clientele using the services corresponds to the clientele initially targeted; if the services delivered correspond to those planned; if the resources planned were effectively made available; and if the delivery procedure corresponds to the one initially defined. Other questions focus on evaluating the process itself and identifying its strengths and weaknesses in order to reinforce or correct some of the elements comprising the new mechanisms and tools. There are three cases (each of the selected areas) analysed using different perspectives (multiple units of analyses). Data are collected from policy-makers, managers and clinicians, as well as clients and informal caregivers using different methods (interviews, focus groups, surveys). Other data are obtained from documentation analysis (minutes from the meetings), participating observation, management data monitoring or clinical files analysis.
Effectiveness is being evaluated using a quasi-experimental design (pre-test, multiple post-tests with control group). A sample of frail older people in the three This article is published in a peer reviewed section of the International Journal of Integrated Care  study areas is followed for 5 years, as is a comparative sample in three comparable areas elsewhere in the Province of Quebec. The variables measured are: functional autonomy, satisfaction in regard to the services received, client empowerment, caregivers' burden, utilization of health and social services, and drug use. An economic analysis is also being performed.
More information on the PRISMA model and the PRISMA-Estrie Project can be found in a previous paper published in the Journal w15x. The present paper reports on one of the study included in the implementation analysis targeting the degree of implementation of the six components of the PRISMA model.

Methods
Based on the PRISMA model, a set of objective and measurable indicators of implementation were generated for each component. Those indicators were fully discussed and approved by two committees acting as focus groups meeting researchers, policy-makers, managers and clinicians involved in the PRISMA pro-ject. The first group is provincial and includes two teams of researchers coming from Laval and Sherbrooke universities interested in ISD systems, policymakers from the Quebec Ministry of Health and Social Services, and managers from five Regional Health and Social Services Authorities. The second is a local group involved in the implementation of the Estrie Project. It includes researchers from the Sherbrooke University, a policy-maker from the Quebec Ministry of Health, managers from the Estrie Regional Health Authority and the health organizations of the three experimental areas, and clinicians. The provincial group met once and the local group twice for discussing this issue. The groups were also asked to weight the relative importance of each indicator and the relative importance of each component. For the latter, it was agreed to assign 20% each to coordination, the single entry point and case management, 15% each to the single assessment tool and computerized clinical chart, and 10% to the individualized service plan. A description of the indicators and their relative weights can be found in Table 1.  assess all the indicators. Coordination was rated according to two sources of data about the different meetings held by the Joint Governing Boards and the Service Coordination Committees. Formal minutes of those meetings were systematically reviewed and were complemented by the notes of one of the research assistants who attended all the meetings using participating observation strategy. The single entry point indicators were completed using statistics from the local agencies where they are based. The number and caseload of case-managers were collected from the coordinators responsible for the implementation of the ISD system in each area. These data were validated by financial information from the Regional Health and Social Services Authority. The charts of all clients involved in the ISD system and referred to case-managers were systematically reviewed to quantify the utilization of the SMAF tool and the individualized service plans. For the computerized clinical chart, we did a monitoring of the equipments in each area and we used data on the real utilization of the system. Table 2 shows the implementation degree for each of the six components and each area. At the beginning of the implementation evaluation study, the coordination process was already well underway in all three areas, especially Sherbrooke where the implementa-tion degree for this component was over 90%. In Sherbrooke, however, there was a broad consultation process and a reform of the ISD governance model during the spring of 2003, which explains why the rate declined to 76% in July 2003. The implementation of the single entry point started in January 2002 in the two rural areas and later (July 2002) in Sherbrooke. The first case managers were hired in January 2002 in Sherbrooke, but there was a slight delay in the rural areas. Consequently, the individualized service plan use started at these times. Implementation of the SMAF tool began before the official start of the study, and the Iso-SMAF case-mix classification system was being used by the local agencies for admitting clients to institutions and by the regional board for financing by July 2002. Finally, the computerized clinical chart was piloted in the Granit Region in July 2002 and extended to the other areas in November 2002. Overall, in July 2003, the ISD system was implemented at the rate of 73%, 71% and 70% in Sherbrooke, Granit and Coaticook, respectively.

Discussion
In the literature on ISD systems, there are few data on the implementation process w4, 16, 17x. Even when it is reported, implementation is mostly summarized very briefly within a paper focusing mainly on the outcome of the intervention. In this study, we attempted to quantify the implementation in order to monitor the introduction of the components of the program under study.
The weighting of the components was arbitrary and based on the opinions of the researchers, clinicians, managers and policy-makers involved in the implementation. There is a hierarchy within these components: coordination is the base without which the other components cannot be implemented. Case management is also essential for implementation of the individualized service plan, the single assessment tool and computerized clinical chart. Therefore, coordination and case management are enabling factors that need to be weighted more than the others. The single entry point was also heavily weighted, because of its importance and the complexity of implementing it, especially in the urban area where multiple public, private and voluntary agencies are geographically scattered in town and must converge on a unique point.
The coordination element was already very highly rated at the beginning of the study and showed a ceiling effect. This suggests that more indicators should have been developed to illustrate higher level of coordination, particularly at the governance level. However, it must be said that discussions around coordination were already well engaged in the region for several years and that the suggested indicators could be appropriate and more modulated in other areas where the coordination process has to be initiated.
The PRISMA research group does not control the actual implementation. This is managed by the Regional Health and Social Services Authority and the local Governing Boards and agencies. Although the implementation of the PRISMA model was supported by a government policy working towards the integration of services for frail older people, it was implemented during a period of health and social services budget constraints. The implementation was delayed and has not yet been completed, mostly because of the slowness of the regional authority to provide funding to create new case manager positions. As noted by Leutz, ''integration costs before it benefits'' w4x. The investment in ISD system should be significant enough to generate some impact and benefit by improving the efficacy and efficiency of the system. We originally expected that the implementation would be completed over an 18-month period. This period has now been extended to 30 months, which is more than double the time anticipated.
The major and most costly component of the PRISMA model is case management, the implementation of which was only half that expected in July 2003. The weighting of this element should probably have been even higher given its importance in the integration of services. Recently, new investments have been announced to complete the staffing of case managers and we anticipate that the implementation percentage of this component should be over 90% by July 2004.
Such an improvement will push the overall implementation degree to over 80% in the three areas. It is probable, although there are no data to support this hypothesis so far, that there is a threshold over which ISD begins to have a significant impact on the health and social services delivery system.
In general, the individualized service plan implementation degree decreased as the caseload of the case managers increased and approached the planned objective of 45 cases. This probably means that this case number is too high and does not allow case managers to complete the plan properly for their clients within an appropriate timeframe. Because of this, and other indicators mentioned in the interviews with case managers, the coordinators are considering decreasing the caseload to 40.
This method of rating the implementation cannot be fully validated, since there is no concurrent method of assessing quantitatively the implementation of a program. The use of focus groups and a well-defined theoretical model confers it a content validity. The relation of the implementation rating with the impact of the system on health services utilization and cost could be used in the future as construct validity. The data were collected and the rating was applied by only one person in the present study, so the interrater reliability is not an issue here. Since there were only three areas and the memory bias would have been very important, it was not possible to check the test-retest reliability of the method.
The data generated by this method have been very useful to managers and policy-makers. It gives them continuous information on the implementation process. It has been also a very powerful tool when arguing for more investments to complete the implementation. In our project, the implementation degree was a very strong incentive for investing more in the ISD system, for creating more case-manager positions, for financing the computerized clinical chart, and for implementing the case-mix classification system. It provided also good evidence for adjusting the case load of case-managers.
This method could also be useful for quantifying the ''dosage'' of the intervention for the outcome study. We may then correlate the implementation degree with the outcome measures relating to the individual, the family or health care utilization. It will help to decrease the ''black box'' effect of health services intervention and to avoid a ''type 3 error'' in measuring the effect of an intervention that has not really been or was not sufficiently implemented. It will be interesting to check if there is a threshold in this percentage over which significant impact may be demonstrated.