Aging is irrevocably accompanied by the loss of physical, mental, and social strength and capabilities [1, 2, 3]. To address the diverse needs of older people for care and support, integrated care programmes (ICPs) are recommended. ICPs aim to provide a continuum of care for older people, where professionals in different domains cooperate and coordinate care taking into account the often complex care needs, and the individual preferences of older people within a broad range of (health and social) services over an extensive timeframe [4, 5].
The Dutch government aims to help people grow old independently in their trusted environment among others by facilitating better support and care at home, support to informal caregivers and volunteers, and more suitable housing for older people at the mean time saving on costs for institutional care . General practitioners (GPs) and their specialised ‘primary care assistant practitioners’ are considered the main providers of complex care for older people. GPs, specialists in geriatric medicine and social geriatricians are partners in geriatric care who, together with pharmacists and home care providers, indicate what structure and resources are required to provide good care . Primary care assistant practitioners are regarded as essential players in identifying care needs of patients, organizing care and coordinating primary care .
This study is part of a larger study, commissioned by the National Health Care Institute. In the first part of this study, an international rapid realist review (RRR)  was conducted with the objective to provide insight into the relationships between the context (C) (wider external items) in which ICPs for community-dwelling older people are applied, the mechanisms (M) (enablers, underlying entities, processes, or structures) by which the ICPs (do not) work, and the outcomes (O) (intended and unintended) resulting from this interaction [14, 15, 16]. As a result, the RRR established an initial programme theory (PT): a hypothesised explanation of how a complex intervention or programme is expected to work [17, 18]. This initial PT demonstrated that it is essential to establish multidisciplinary teams (C) of competent healthcare providers (HCPs) (C) in order for them to provide person-centred care (M) and involve older people and their informal care giver(s) in the care process (M). This has a positive effect on the functionality of older people (O), hospital-related outcomes (O), and the quality of life of older people (O). Also, by means of a multidisciplinary core team (C) a strong collaboration within and between disciplines can be established (M), which has shown to increase the satisfaction levels of older people, informal caregivers and HCPs (O). Next to efficient use of information technology (C), organisational alignment (C) on all levels, and the provision of sufficient financial resources (C) it is important that that training and education of HCPs (C) in e.g. communication skills takes place, for them to communicate effectively with all involved stakeholders (M). This can result in a delayed placement of the older person in a nursing home (O), reduced use of healthcare services (O), and reduced healthcare costs (O).
The environment plays an important role in the development and implementation of ICPs [19, 20, 21]. Too often ICPs lack a theoretical underpinning and hence have been accused to jump to solutions and to do more harm than good . What may successfully work in one setting regarding ICPs, may not work in a different setting. Ideally, the appropriate combination of components of an ICP, needs to be developed and based on the values and preferences of the local setting, however this remains underexposed in the literature [19, 20, 21]. So, notwithstanding the insights derived from the RRR, as mentioned previously, ultimately ICPs need to fit the local setting. The National Health Care Institute in the Netherlands indicated that, ideally, existing, often incidentally developed, local care initiatives should be replaced by conceptualised ICPs for community-dwelling older people. However, different stakeholders in different parts of the Netherlands may hold different beliefs about why, how and for whom an ICP may result in what outcomes and when.
As the RRR provided an international perspective on the items that play an important role in ICPs for older people, but ICPs need to fit the local setting, it is essential to determine which items are relevant for the Dutch setting. Based on that, stakeholders are provided with evidence and practical guidance to establish effective ICPs. This can help to reduce the degree of heterogeneity present in outcomes of ICPs. The aim of this study is to assess consensus on the relevance of context items, mechanisms, programme-activities, and outcomes of ICPs for community-dwelling frail older people for the Dutch setting by various stakeholders and to refine the PT for the Dutch setting, where appropriate.
An e-modified Delphi study was conducted to assess consensus for the Dutch setting on the context items, mechanisms, programme-activities and outcomes of ICPs for community-dwelling older people, which emerged from the RRR previously conducted . A Delphi study consists of multiple rounds in which data are collected by sending out a questionnaire that needs to be filled in by a panel of experts on a particular topic. The anonymous responses are aggregated and shared with the panel after each round in the form of a group result [19, 24]. In a classical Delphi study, the aim is to elicit opinion and gain consensus, may consist of three or more rounds, and has an open qualitative first round which allows Delphi panel experts to record responses. In this study, the term ‘modified’ refers to a Delphi study that consisted of two rounds, and where in round 1 Delphi panel experts were provided with items of the RRR, of which they are requested to assess their relevance for the Dutch setting [25, 26]. A Delphi study is an efficient method for obtaining valuable input from multiple experts in a relatively short timeframe and clarifies which items are more/less relevant and why, or which items are missing from the theory presented, in this case the RRR. Information on consensus among experts is particularly useful in the process of refining the PT and explaining why integrated care does (not) work for (frail) older people, how, and in this specific context.
A purposive sampling strategy was used to identify experts with relevant experience in the field of integrated care for older people, aiming for diversity regarding age, gender, profession, and the setting of the ICP(s) they were involved in. In order for the experts to be selected for the Delphi panel, they needed to be actively involved in the implementation of programmes regarding integrated care for (frail) older people at home which were being monitored or evaluated in the Netherlands. Their active involvement in the implementation of ICPs depended on their role as e.g. researchers, healthcare providers, policy advisors, managers etc. Participants for the Delphi expert panel were recruited across the Netherlands through the professional networks of various parties involved in this study, i.e. the commissioner of the current study, a steering committee established for the larger study (see Acknowledgements) and, the researchers of the current study. Experts who met the selection criteria were invited by email with information about the study objectives and details of the Delphi study. Those who gave informed consent were included in the study.
Participants were sent an electronic questionnaire via a weblink (SurveyMonkey). The questionnaire started with an introduction of the study, an explanation of the objectives, the structure of the questionnaire, and the definitions of the constructs: context, mechanisms, programme-activities, and outcomes. The questionnaire continued with six general questions regarding gender, age, highest level of education, current job position, number of years working within the position, and number of years of experience with integrated care for older people. The questionnaire contained another 71 questions related to ICPs . Participants were asked to indicate the relevance of 15 context items, 14 mechanisms, 20 programme-activities and 22 outcomes. Relevance was measured on a 9-point Likert scale (1 = very irrelevant, 9 = very relevant), with scores 1–3 considered as irrelevant, 4–6 as equivocal/ambiguous and, 7–9 as relevant. Context items included e.g. offering training and education to healthcare professionals, and having organisation support and coordination on all levels; mechanisms included e.g. involvement of older people and informal caregivers, and having effective communication between all stakeholders, programme-activities included e.g. performing comprehensive geriatric assessments, and deployment of case management; and outcomes included e.g. delayed move to nursing home, and quality of life (see Appendix A for the complete list). The questionnaire ended with two open questions. In this part, participants were able to provide additions to the context items, mechanisms, programme-activities, and/or outcomes in the questionnaire. The participants were also asked for general comments/suggestions about items and the questionnaire itself. Data collection of round 1 took a total of two weeks.
In the second Delphi round, items on which dissensus was found during the first Delphi, were included. The questionnaire started with the same general questions as round 1. Subsequently, participants were asked to reassess the relevance of the context items, mechanisms, programme-activities, and outcomes on a 9-point Likert scale. At the end of the questionnaire, participants were asked for general comments/suggestions on the items and the questionnaire. During the second round, participants were shown a summary of the group results from the first Delphi round, including 1) the median assessment results and interquartile range (IQR) on each item, 2) the level of (insufficient) consensus between the participants and, 3) whether consensus achieved. The IQR is the difference between the 3rd and 1st quartile in which 50% of core values lie . The IQR also shows the degree of convergence of the answers [28, 29, 30, 31]. A summary of the group results were shown to give insight into the level of (dis)agreement between experts in the first round and to generate additional insights about the specific item(s). It has been shown that providing feedback regarding the level of group agreement reached, influences the achievement of level of consensus subsequently . Data collection of round 2 took a total of two weeks.
The measures concerning the operationalization of the level of consensus among participants were determined in advance . In the literature, no standard threshold for consensus is offered , with thresholds for consensus ranging from 55%–100% . In this study, the 9-point scale was categorized into three ranges: 1–3 as irrelevant; 4–6 as equivocal; and 7–9 as relevant. The cut-off point for consensus among panel members was set on 75% [34, 36, 37], including the condition that less than 15% of the panel needed to have a scoring in the 1–3 range [38, 39]. All items with scores in the 4–6 range and without consensus, were presented again to the expert panel in Delphi round 2. Table 1 demonstrates when an item was defined as irrelevant, equivocal, or relevant based on the overall median panel score in both rounds. The degree of consensus of the respondents on each context item, mechanism, programme-activity, and outcome was analysed based on the median scores of the group. Only fully completed questionnaires in both rounds were included in the analyses. The analyses were performed in MS Excel.
|OVERALL PANEL MEDIAN IN 1–3 POINT RANGE||OVERALL PANEL MEDIAN IN 4–6 POINT RANGE||OVERALL PANEL MEDIAN IN 7–9 POINT RANGE|
|Round 1||Dissensus (<75%)||Equivocal → included in round 2||Equivocal → included in round 2||Equivocal → included in round 2|
|Consensus (≥75%)||Irrelevant||Equivocal → included in round 2||Relevant|
|Round 2||Dissensus (<75%)||Equivocal||Equivocal||Equivocal|
Based on the findings of the Delphi study, the PT presented in the RRR was adjusted where appropriate. Consensus on items being relevant, remained part of the PT or were added to the PT. Consensus on items being irrelevant or no consensus on items were removed from the PT.
As this study does not involve patients or study subjects, according to the Dutch Medical Research in Human Subjects Act (WMO) in the Netherlands, an ethical approval was not needed. However, all participants provided their consent and participation in the survey was anonymous.
A total of 35 people was approached to participate in the Delphi study, of which 21 people agreed (Figure 1). One person mentioned she did not have the time to participate, whereas the other 13 did not respond to our invitation and thus did not provide a reason not to participate. Of the 21 participants, three did not fully complete the questionnaire in round one (completion rate = 86%), and one in round two (completion rate = 94%). One participant in round one mentioned she found the questions too hard to interpret. Other participants did not provide a reason for not completing the questionnaire. The final data analyses included responses of 17 participants. In Table 2 the characteristics of the participants are shown.
|CHARACTERISTIC||PARTICIPANTS (N = 17)|
|Average (SD)||52.5 (7.3)|
|Highest level of education (%)||Bachelor||12|
|Years of experience||Min-Max||3–35|
|Average (SD)||16.4 (10.8)|
The results on each of the context items, mechanisms, programme-activities, and outcomes of the first and second Delphi round are shown in Appendix A and B respectively. Table 3 shows the total number of items per category (context, mechanisms, programme-activities and outcomes), the number of items on which consensus was achieved, and how many items were equivocal in both two rounds.
|CATEGORY||TOTAL NUMBER OF ITEMS||ITEMS FOUND RELEVANT (CONSENSUS), N (%)||ITEMS UNDECIDED (EQUIVOCAL), N (%)|
|Total||71||51 (72)||20 (28)|
|Context||15||12 (80)||3 (20)|
|Mechanisms||14||12 (86)||2 (14)|
|Programme-activities||20||12 (60)||8 (40)|
|Outcomes||22||15 (68)||7 (32)|
|Total||20||6 (30)||14 (70)|
|Context||3||2 (67)||1 (33)|
|Mechanisms||2||1 (50)||1 (50)|
|Programme-activities||8||2 (25)||6 (75)|
|Outcomes||7||1 (14)||6 (86)|
In the first round, 51 of the total 71 items were considered relevant and consensus was achieved among the experts. The overall median rating by the experts was between the 7–9 point range with a consensus level over 75%. Twenty items remained undecided and were considered equivocal. The panel median for these items was in the 4–6 point range (eight items) and 7–9 point range (12 items) with consensus lower than 75% within the same 3-point region. Consensus on items being found relevant among the experts was observed in the mechanisms (86% of the items), followed by the context items (80% of the items), the outcomes, (68% of the items), and programme-activities (60% of the items). Experts did not propose any additional items to include in round 2. In round 2, two participants indicated that the outcome factor ‘well-being of older person’ was missing in the questionnaire.
In the second round, the 20 items that were rated as equivocal in round one were included. Of these items, consensus on six items was found among the experts. The overall median of the experts was between the 7–9 point range with a consensus level over 75%. For 14 items (70%) the degree of relevance remained undecided (equivocal). The overall median of the experts was in the 4–6 point range (three items) and in the 7–9 point range (11 items). Consensus on items being relevant among the experts was observed for the context items (67% of the items), followed by the mechanisms (50% of the items), programme-activities (25% of the items) and the outcomes (14% of the items). After two rounds, for 57 items of the total 71 items consensus was achieved among the experts whereby all the items were considered relevant.
The context items that were found to be highly relevant for the Netherlands among the Delphi panel experts concern organizational support and coordination, (financial) resources and incentives to invest in integrated care for frail older people, the alignment of existing health and social care systems, and the smart use of (information) technologies. Experts agreed that training of professionals to be competent and highly skilled and having a multidisciplinary core team were context items of great importance. The integration of case management in a broader program or the healthcare system, as well as, offering guidance and support to older people and informal caregivers was also considered relevant. In light of the context item ‘offering remunerative and financial support’, experts mentioned that the financing of integrated care for older people should be led by patients’ needs. This entails that financing should not only be restricted to medical care, but also include social care and support for caregivers.
The context items for which insufficient consensus was found after two rounds concerned the degree of integration of Advanced Practice Nurses (APN) in the healthcare system, for which no reason was provided.
Among the Delphi panel experts there was agreement on the relevance of involving older people and their caregiver(s) (e.g. in shared decision-making, developing care plans) and of the importance of establishing a good relationship between the older person and the HCPs as mechanisms. Communicating effectively plays an essential role, not only between HCPs to ensure optimal interprofessional collaboration, but also between HCPs and older people and their informal caregivers. Enabling collaboration structures were considered very relevant for optimal functioning of a fully integrated interprofessional care team. Also, the HCPs need to provide person-centred care by putting the older person central and focus on the needs, preferences, and possibilities of the individual.
Delphi panellists disagreed on the relevance of focusing on system goals (e.g. improved national system integration) for ICPs. A reason for not finding this mechanism relevant, was not provided.
Experts considered the programme-activities identification and selection of the right target group, incorporating risk prevention in ICPs, performing comprehensive geriatric (home) assessments, and frequent (preventive) home visits to be highly relevant for the Dutch situation. Various care activities, such as the development and implementation of individual care plans, setting up a hospital discharge plan, medication adjustment and alignment (e.g. at care transition) were also considered programme-activities of high relevance. Supporting self-management of older people, the provision of case management, as well as empowerment of patients were found relevant too.
The Delphi rounds also demonstrated that the degree of relevance was undecided for multiple programme-activities, such as the generic and disease-specific deployment of APNs, performing (telephone) follow-up appointments, having specialized clinics regarding memory/dementia care in primary care, standardization of processes, and the use of information technology (IT) for risk inventory and reminders. However, a reason on disagreement was not provided.
A high degree of relevance was found for increased functionality, improved self-management of the older person, quality of life (mixed results in the literature), improved (perceived) health, decreased decline in mental health (e.g. depression), a higher satisfaction of the patient, informal caregiver(s) and HCP(s). Moreover, the possibility for the older person to stay longer at home, and hospital-related outcomes (mixed results in the literature) were assessed to be of high relevance. Experts also agreed with high relevance being found on the outcomes of use of hospital services/health system (mixed results in the literature), improved access to healthcare and social care, and improved use of case management services.
Dissensus on relevance has been found concerning the following outcome measures: increase in the performance frequency of early detection screening tests for certain conditions (e.g. diabetes, hypertension, vitamin B12 deficiency) and immunizations (e.g. influenza vaccinations) due to the highly clinical nature of the outcomes; reduced medication use by older people, improved timeliness of communication (e.g. to primary care), cost-effectiveness, and mortality. A reason for disagreement on the latter four outcomes was not found.
Based on the findings of the Delphi rounds, the PT was refined for the Dutch setting. In Box 1 this refined PT is shown, with the items that were added, being underlined.
Box 1 Refined PT on ICPs for community-dwelling frail older people in the Dutch setting.
C = context; PA = programme-activity; M = mechanism; O = outcome
Considering the increase in the number of older people (C) and decrease in access to hospital beds (C), ICPs with well-designed implementation processes (C) offering continuity of care (PA) are needed. The national and local governments can play a role in facilitating (components of) ICPs by promotion via funding or policy (C) and by providing clarity on legislation and regulations concerning ICPs (C). By means of case finding the right patient population is identified and selected (PA) to deliver the right care at the right time. It is essential to establish well-skilled (C) multidisciplinary teams of competent HCPs (C) providing person-centred care (M) and self-management support (PA) and making sure that patients are empowered (PA) to achieve good health. HCPs need to work closely together (M) and communicate effectively with stakeholders from other domains e.g. primary care, secondary care, community care, social/policy domain, and also informal caregivers (M). By means of education (C) and involving older people and informal caregivers in the care process (M), and trusting the general practitioner (M) and/or the primary HCP (e.g. home visiting professional) (M) a strong relationship between them and the HCP’s (M) should be built. This way management and monitoring of care activities (M, PA) can be optimized with having a clear portfolio of patients (C) whereby continuous feedback to HCP’s (M) needs to be provided. Several programme-activities may contribute to achieving the desired results, such as conducting extensive geriatric assessments/shared assessment processes (PA), setting up individual care plans (PA), having (preventive) home visits (PA), performing case management (PA), managing medication treatment (PA), hospital discharge planning (PA). Next to the alignment of health and social care systems and organizations (C), financial support (C) with e.g. incentives for active participation (M), efficient use of information technology (C), and integration of case management in ICPs (C) emerged also as key elements. ICPs demonstrate positive effects on the functionality (O), mental health (O), self-management skills (O), perceived health (O) of older people, hospital-related outcomes (O), quality of life (O), use of healthcare services including case management (O), and their access to healthcare and social care (O). Besides improved care processes (O), end-of-life discussions were increased (O), the burden on informal caregiver(s) was reduced (O), and there was a delayed placement in a nursing home (O) improving the satisfaction of older people, informal caregivers and HCPs with the care provided (O).
In this study we aimed to refine the PT for ICPs for community-dwelling frail older people for the Dutch setting by providing insight into the level of consensus on the relevance of context items, mechanisms, programme-activities, and outcomes identified in the RRR. Based on two Delphi rounds, consensus was reached on a set of 57 out of 71 items (80%) of the initial PT, derived from a previous conducted RRR using international literature. Based on the findings of the Delphi study, the initial PT was extended. The added items in the refined PT included increase in the number of older people, decrease in access to hospital beds, well-designed ICP implementation processes, case management, having a clear portfolio of patients, the role of the national/regional governments, aligning existing health- and social care systems, management and monitoring of care activities, strong relationship between older person and HCPs with patients putting their trust in GP, providing continuous feedback to HCPs. These added outcomes were self-management, perceived and mental health, burden on informal caregiver(s), frequency of end-of-life discussions, healthcare access, and care processes. In the refined PT the items ‘having follow-up appointments’ (programme-activity) and ‘healthcare costs/cost-effectiveness’ (outcome) were removed. Also, not finding consensus on the relevance concerning the inclusiveness of APN may illustrate that there is unclarity about the APN role as part of ICPs in the Netherlands. As nowadays primary care assistant practitioners play an important role in primary care for older people, the main role of APNs in ICPs is not fulfilled like before. The main role of APNs seems to have changed over time from practitioner to consultant [12, 40].
When comparing our findings with those of other studies, it must be noted that there are not many Delphi studies on integrated care specifically for older people. Briggs et al. (2018) generated consensus on the actions required to implement the World Health Organization Integrated Care for Older People (ICOPE) approach . In line with our study, consensus was found on setting up individualised interdisciplinary care plans for patients, active case finding, incorporating prevention programmes, performing geriatric assessments, care delivery by interdisciplinary teams, educational support for formal and informal carers, and the use of data sharing platforms . Items on which no consensus was found by Briggs et al. (2018), were the use of provider report cards, traditional and complementary medicines, and the development of new work cadres . Zonneveld et al. (2020) investigated the values that underpin integrated health services delivery and found consensus on values such as ‘person-centred’, ‘co-produced’, ‘collaborative’, ‘preventative’, and ‘co-ordinated’, comparable to our findings . Regarding values related to IC, no consensus was found on ‘sustainable’, ‘innovative’, ‘proficient’, ‘safe’, and ‘realistic’ due to not being specific or essential enough for IC . They, however, did not focus on programmes for older people specifically.
The strength of the current study lies in the use of the structured, electronic Delphi technique to further refine the PT in our RRR and explaining why IC does (not) work for (frail) older people, how, and in what context. To the best of our knowledge, we are one of the first to opt for a Delphi study following a RRR, whereas often individual interviews are conducted. Given the scarcity of resources, this appears to be an efficient method for obtaining meaningful input from multiple experts in a relatively short timeframe. This method makes it clear which items are more/less relevant, and/or which items are missing from a RRR and why items are considered less relevant.
However, a few limitations need to be considered for this Delphi study. The first one being the size of the Delphi panel. We invited 35 experts to participate in the Delphi study, but not all responded to our invitation. Nonetheless, sufficient diversity in the Delphi panel was achieved, which is considered more important in terms of validity of study findings. Currently, there are no universally agreed criteria for the selection of experts, and the minimum or maximum number of experts on a panel [34, 43]. A second limitation concerns the formulation of questions in the survey. Delphi panel members indicated that several questions were open for own interpretation and could be explained in more detail. The high level of consensus reached after two rounds given a diverse panel, is however very encouraging. To clarify questions in the second Delphi round, some were slightly reformulated or a brief explanation was included. A third limitation relates to the e-interaction between panel members. Exchange of arguments between experts and the authors was only possible digitally, which has hindered in not or partially being able to explain the lack of consensus. In order to acquire more information on the reasoning of members, a blended or ‘physical’ Delphi study could be more suitable.
The findings of this study can be valuable for both HCPs, policymakers and researchers involved in the development, implementation and/or evaluation of ICPs for older people. Considering the interrelatedness of items, it is suggested to collectively implement the items mentioned in this study, to increase the effectiveness of ICPs. Developing a network in which various stakeholders (e.g. general practitioners, primary care assistant practitioners, pharmacists, community nurses, informal caregivers and older people) have good partnerships, can ensure a better connection between provided services and the needs and preferences of older people. Any forms of consultation to structurally exchange knowledge and expertise may support the (complex) care demand of individual older people. In an ideal situation, tailor-made interventions are offered depending on the different degrees of level of frailty of the older person. In addition, it is important to provide for a systematic risk inventory (e.g. by means of information technology), in which older people at risk are identified in an early stage and subsequently proactive policy can be pursued from the network.
As in this study the context items, mechanisms, programme-activities, and outcomes have been assessed on their relevance, a next step in further research would be to see to what extent these have been implemented and reached the intended outcomes within their context. Also, items on which no consensus was found need to be further investigated on the reason behind it and to explore whether ICPs in the Netherlands are conceptually different than elsewhere or not. Additionally, further validation of context items, mechanisms, programme-activities, and outcomes needs to take place by involving the older people, informal caregivers, and federations for older people/patients [44, 45].
In this study, consensus within the Delphi panel was reached on a set of 57 out of 71 items (80%) based on two Delphi rounds, with items being found relevant. Based on the findings of the Delphi study, the PT for ICPs for older people in the Dutch setting was refined. The added items in the refined PT included increasing number of older people, decreasing access to hospital beds, well-designed ICP implementation processes, case management, having a clear portfolio of patients, the role of the national/regional governments, aligning existing health- and social care systems, management and monitoring of care activities, strong relationship between older person and HCPs with patients putting their trust in GP, providing continuous feedback to HCPs. Further validation of context items, mechanisms, programme-activities, and outcomes needs to take place by involving the older people, informal caregivers, and federations for older people/patients. Additionally, items on which no consensus was found, need to be further investigated on the reason behind it.
Our gratitude goes out to the steering committee of this study: Dr. Isabelle Fabbricotti (Erasmus School of Health Policy and Management, Erasmus University Rotterdam, Rotterdam, The Netherlands); Dr. Robbert Gobbens (Faculty of Health, Sports and Social Work, Inholland University of Applied Sciences, Amsterdam, the Netherlands; Zonnehuisgroep Amstelland, Amstelveen, the Netherlands; Department of Primary and Interdisciplinary Care, Faculty of Medicine and Health Sciences, University of Antwerp, Antwerp, Belgium); Dr. Nick Goodwin (Faculty of Health and Medicine, University of Newcastle, New South Wales, Australia); Dr. Pim Valentijn (Essenburgh, Hierden, The Netherlands).
We also would like to thank the team of the National Health Care Institute (Diemen, The Netherlands) for their collaboration throughout the research: Marjolein de Booys MSc, Carel Mastenbroek MSc, Aster de Lange RN, and Tjitske Vreugdenhil MD.
Milawaty Nurjono, Health Services Research, Changi General Hospital, Singapore.
Ronald Uittenbroek PhD., Head of school, school of nursing, Windesheim university of applied sciences, Health and social care dept., Netherlands.
The authors have no competing interests to declare.
Laidlaw K, Pachana NA. Aging, mental health, and demographic change: Challenges for psychotherapists. Prof Psychol Res Pr. 2009; 40(6): 601–608. DOI: https://doi.org/10.1037/a0017215
Milanović Z, Pantelić S, Trajković N, Sporiš G, Kostić R, James N. Age-related decrease in physical activity and functional fitness among elderly men and women. Clin Interv Aging. 2013; 8: 549–556. DOI: https://doi.org/10.2147/CIA.S44112
Mendes De Leon CF. Social engagement and successful aging. Eur J Ageing. 2005; 2(1): 64–66. DOI: https://doi.org/10.1007/s10433-005-0020-y
Janse B, Huijsman R, de Kuyper RDM, Fabbricotti I. Do integrated care structures foster processes of integration? A quasi-experimental study in frail elderly care from the professional perspective. Int J Qual Health Care. 2016; 28(3): 376–83. DOI: https://doi.org/10.1093/intqhc/mzw045
Mur-Veerman I, Hardy B, Steenbergen M, Wistow G. Development of integrated care in England and the Netherlands: Managing across public–private boundaries. Health Policy. 2003; 65(3): 227–241. DOI: https://doi.org/10.1016/S0168-8510(02)00215-4
Hoogendijk EO. How effective is integrated care for community-dwelling frail older people? The case of the Netherlands. Age and Ageing. 2016; 45(5): 585–588. DOI: https://doi.org/10.1093/ageing/afw081
Threapleton DE, Chung RY, Wong SYS, Wong E, Chau P, Woo J, et al. Integrated care for older populations and its implementation facilitators and barriers: a rapid scoping review. Int J Qual Health Care. 2017; 29(3): 327–334. DOI: https://doi.org/10.1093/intqhc/mzx041
Briggs AM, Valentijn PP, Thiyagaranjan JA, Araujo de Carvalho I. Elements of integrated care approaches for older people: a review of reviews. BMJ Open. 2018; 8(4): e021194. DOI: https://doi.org/10.1136/bmjopen-2017-021194
Looman WM, Huijsman R, Fabricotti IN. The (cost-)effectiveness of preventive, integrated care for community-dwelling frail older people: a systematic review. Health Soc Care Community. 2019; 27(1): 1–30. DOI: https://doi.org/10.1111/hsc.12571
Rijksoverheid. Zorg en ondersteuning thuis. [Care and support at home] [Webpage on the internet]. 2020 [Cited 2020, December 28]. Available from: https://www.rijksoverheid.nl/onderwerpen/zorg-en-ondersteuning-thuis/langer-zelfstandig-wonen. [in Dutch].
van de Rijdt-van de Ven AHJ. Complexe ouderenzorg in verzorgingshuis en thuis. Handreiking, samenhangende zorg in de eerstelijn. [Complex care for older people in care homes and at home. Guide, coherent care in primary care]. Utrecht, the Netherlands; 2009. [in Dutch]
De Groot K, de eer A, Versteeg S, Francke A. Het organiseren van langdurige zorg en ondersteuning voor thuiswonende patiënten. Ervaringen van praktijkondersteuners in de huisartsenzorg. [Organizing long-term care and support for patients living at home. Experiences of assistant practitioners in primary care] Utrecht, the Netherlands; 2018. [in Dutch]
Ahmed A, Van den Muijsenbergh METC, Mewes JC, Wodchis WP, Vrijhoef HJM. Untangling the inter-relatedness within integrated care programmes for community-dwelling frail older people:a rapid realist review. BMJ Open. 2021; 11: e043280. DOI: https://doi.org/10.1136/bmjopen-2020-043280
Kodner DL. All together now: a conceptual exploration of integrated care. Healthc Q. 2009; 13: 6–15. DOI: https://doi.org/10.12927/hcq.2009.21091
Elissen AMJ, Steuten LMG, Lemmens LC, Drewes HW, Lemmens KMM, Meeuwissen JAC, et al. Meta-analysis of the effectiveness of chronic care management for diabetes: investigating heterogeneity in outcomes. J Eval Clin Pract. 2013; 19: 753–62. DOI: https://doi.org/10.1111/j.1365-2753.2012.01817.x
Drewes HW, Steuten LMG, Lemmens LC, Baan CA, Boshuizen HC, Elissen AMJ, et al. The effectiveness of chronic care management for heart failure: meta-regression analyses to explain the heterogeneity in outcomes. Health Serv Res. 2012; 47: 1926–59. DOI: https://doi.org/10.1111/j.1475-6773.2012.01396.x
Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review – A new method of systematic review designed for complex policy interventions. J Health Serv Res Policy. 2005; 10(1): 21–34. DOI: https://doi.org/10.1258/1355819054308530
Mays N, Smith J. What can England’s NHS learn from Canterbury New Zealand? BMJ. 2013; 347: f6513. DOI: https://doi.org/10.1136/bmj.f6513
World Health Organization. Framework on integrated, people-centred health services. [Webpage on the internet]. 2016 [Cited 2020, December 28]. Available from: http://apps.who.int/gb/ebwha/pdf_files/WHA69/A69_39-en.pdf?ua=1&ua=1.
Evans JM, Grudniewicz A, Baker GR Wodchis WP. Organizational Context and Capabilities for Integrating Care: A Framework for Improvement. Int J Integr Care. 2016; 16(3): 15. DOI: https://doi.org/10.5334/ijic.2416
Vrijhoef HJM. Towards seamless health care: no theory, no glory. Int J Care Coord. 2019; 22(3–4): 107–108. DOI: https://doi.org/10.1177/2053434519897146
Keeney S, Hasson F, McKenna H. The Delphi Technique in Nursing and Health Research. Oxford: Wiley-Blackwell; 2011. DOI: https://doi.org/10.1002/9781444392029
Hasson F, Keeney S. Enhancing rigour in the Delphi technique research. Technol Forecast Soc Change. 2011; 78: 1695–704. DOI: https://doi.org/10.1016/j.techfore.2011.04.005
De Vet E, Brug J, de Nooijer J, Dijkstra A, de Vries NK. Determinants of forward stage transitions: A Delphi study. Health Educ. Res. 2005; 20: 195–205. DOI: https://doi.org/10.1093/her/cyg111
Okoli C, Pawlowski SD. The Delphi method as a research tool: an example, design considerations and applications. Inf. Manag. 2004; 42(1): 15–29. DOI: https://doi.org/10.1016/j.im.2003.11.002
von der Gracht HA, Darkow IL. Scenarios for the logistics services industry: A Delphi-based analysis for 2025. Int J Prod Econ. 2010; 127: 46–59. DOI: https://doi.org/10.1016/j.ijpe.2010.04.013
Warth J, von der Gracht HA, Darkow IL. A dissent-based approach for multi-stakeholder scenario development – the future of electric drive vehicles. Technol Forecast Soc Change. 2013; 80(4): 566–583. DOI: https://doi.org/10.1016/j.techfore.2012.04.005
Ray PK, Sahu S. Productivity management in India: A Delphi study. Int. J. Oper. Prod. Manag. 1990; 10(5): 25–51. DOI: https://doi.org/10.1108/01443579010005245
Barrios M, Guilera G, Nuno L, Gomez-Benito J. Consensus in the delphi method: What makes a decision change? Technological Forecasting and Social Change. 2021; 163: 120484. DOI: https://doi.org/10.1016/j.techfore.2020.120484
Von der Gracht HA. Consensus measurement in Delphi studies. Review and implications for future quality assurance. Technol Forecast Soc Change. 2012; 79(8): 1525–1536. DOI: https://doi.org/10.1016/j.techfore.2012.04.013
Keeney S, Hasson F, McKenna H. Consulting the oracle: Ten lessons from using the Delphi technique in nursing research. J Adv Nurs. 2006; 53(2): 205–212. DOI: https://doi.org/10.1111/j.1365-2648.2006.03716.x
Williams PL, Webb C. The Delphi technique: A methodological discussion. J Adv Nurs. 1994; 19(1): 180–186. DOI: https://doi.org/10.1111/j.1365-2648.1994.tb01066.x
Kilroy D, Driscoll P. Determination of required anatomical knowledge for clinical practice in emergency medicine: National curriculum planning using a modified Delphi technique. Emerg Med J. 2006; 23(9): 693–696. DOI: https://doi.org/10.1136/emj.2006.037309
Syed A, Hjarnoe L, Aro A. The Delphi Technique In Developing International Health Policies: Experience From The SARSControl Project. Internet J Health. 2008; 8(2): 1–9. DOI: https://doi.org/10.5580/c7d
Waters AM, Smith CT, Young B, et al. The CONSENSUS study: Protocol for a mixed methods study to establish which outcomes should be included in a core outcome set for oropharyngeal cancer. Trials. 2014; 15: 168. DOI: https://doi.org/10.1186/1745-6215-15-168
Gerritsen A, Jacobs M, Henselmans I, van Hattum J, Efficace F, Creemers G, et al. Developing a core set of patient-reported outcomes in pancreatic cancer: A Delphi survey. Eur J Cancer. 2016; 57: 68–77. DOI: https://doi.org/10.1016/j.ejca.2016.01.001
van Hassel D, Baenburg R, van der Velden L. Praktijkondersteuners (POH’s) in beeld: Aantallen, kenmerken en geografische spreiding in Nederland. [primary care assistant practitioners (POH’s): Numbers, characteristics and geographic distribution in the Netherlands] Utrecht, the Netherlands; 2016. [in Dutch]
Briggs AM, Araujo de Carvalho I. Actions required to implement integrated care for older people in the community using the World Health Organization’s ICOPE approach: A global Delphi consensus study. PLoS ONE. 2018; 13(10): e0205533. DOI: https://doi.org/10.1371/journal.pone.0205533
Zonneveld N, Raab J, Minkman MMN. Towards a values framework for integrated health services: an international Delphi study. BMC Health Serv Res. 2020; 20: 224. DOI: https://doi.org/10.1186/s12913-020-5008-y
Powell C. The Delphi Technique: Myths and realities. J Adv Nurs. 2003; 41(4): 376–382. DOI: https://doi.org/10.1046/j.1365-2648.2003.02537.x
Hussler C, Muller P, Ronde P. Is diversity in Delphi panelist groups useful? Evidence from a French forecasting exercise on the future of nuclear energy. Technol. Forecast. Soc. Chang. 2011; 78: 1642–1653. DOI: https://doi.org/10.1016/j.techfore.2011.07.008
Grol SM, Molleman GRM, Wensing M, Kuijpers A, Scholte JK, van den Muijsenbergh, et al. Professional Care Networks of frail older people: an explorative survey study from the patient perspective. Int J Integr Care. 2020; 20(1): 1–13. DOI: https://doi.org/10.5334/ijic.4721