Managing illness is difficult enough without having to navigate poorly integrated health care delivery systems. Yet many patients report that this is precisely the challenge they face . Patients expect their care to be provided by teams of well-integrated health professionals  but research shows that many patients experience fragmentation, poor coordination, lack of information and confusion [3–5]. Patients also face the cost and burden of attending multiple clinics, often with marginal benefit, as well as duplication of effort, poor communication and conflicting advice [6, 7]. Given the increasing number of people with multiple long-term conditions requiring ongoing, coordinated care , the lack of integration in health care delivery has become one of the most urgent problems facing both systems and patients. Moreover, a recent bibliometric analysis demonstrated that the need to provide integrated care for patients has become a central health policy concern .
Routinely monitoring integration in health care delivery from the patient perspective is a critical step in understanding and improving both the patient experience and the quality and efficiency of health care. Yet, two key factors have impeded progress towards this goal. First, there has long been a lack of conceptual clarity in the field of integration in health care delivery [10–12]. A multitude of terms including integration of care, care coordination, continuity of care, collaborative care and joined-up care have been used, often interchangeably, generating confusion and ambiguity. At the same time, many competing frameworks and models of integration have been proposed. While most are multidimensional, they differ in emphasis from the seamlessness of care provision within a single health care setting to the effectiveness of communication and cooperation across a multi-site network of providers or the continuous provision of care by the same individual health care provider [13, 14]. Second, researchers have largely failed to approach the issue of integration in health care delivery in a patient-centric way . Historically, the dominant perspective underlying conceptualisations of integration has been top-down rather than bottom-up, giving rise to both frameworks and assessment approaches that may not emphasise concepts that reflect the problems that patients commonly experience.
In spite of the barriers described above, recent years have seen a burgeoning of instruments developed to measure integration in health care delivery (and equivalent concepts) from the patient perspective. Uijen and colleagues undertook a systematic review of instruments published between 2005 and 2011, identifying 17 patient-reported measures . Our recent search of the subsequently published literature identified three more patient-reported measures [16–18], for a total of 20 measures. However, notwithstanding their value for use in research, these existing measures have limited utility for the routine monitoring of integration in health care delivery. Sixteen of the 20 measures were created for patients with specific clinical conditions or in specific care settings. The specificity of these measures undermines their wide applicability and precludes their use for cross-institutional comparison and learning. Additionally, many of the measures have a significant number of items and appear unnecessarily burdensome to patients, despite growing evidence that brief health-related measures can perform as well as more comprehensive measures . The average number of items in the reviewed measures was 31, with a range of 12–129, hindering their administration at scale.
Given the limitations of current measures, we sought to develop a new patient-reported measure of integration in health care delivery in partnership with end users. Our specific objective was to develop an instrument that is brief, generic, valid and reliable, and that assesses aspects of health care delivery that are both salient to patients and readily modifiable. We intended for the measure to be suitable for all individuals who have recently interacted with two or more people while receiving health care, whether those people were members of an identified health care team within a single setting or members of a multi-site provider network. In this paper, we describe the process of developing and refining survey items through a series of cognitive interviews, an established method to ensure item interpretability , and verifying the interpretability of the items through a pilot administration of the final instrument.
Drawing on both existing conceptualisations of integrated care [13–15] and the findings of research examining patients’ experiences and perspectives [7, 21–24), we chose to focus on four distinct domains of integration. The first domain, information sharing, represents effective information transfer across members of the health care team. The second domain, consistent advice, represents concordant information provision by members of the health care team. The third domain, mutualrespect, represents respect and collaboration among members of the health care team. The final domain, role clarity, represents patient understanding of the different roles of the various members of their health care team. These domains are not specific either to integration within an identified health care team in a single setting or to integration of those working in a network across multiple settings.
We formulated a pool of candidate items for each domain, focusing purposefully on negatively worded items that assessed the frequency of poor care integration, based on the premise that patients may more readily recall examples of poor quality care. For information sharing, we focused on items assessing how often the patient was required to do otherwise unnecessary or additional work because of a deficit in information transfer by members of the health care team. For consistent advice, we focused on items assessing how often the patient experienced confusion due to receipt of conflicting information or advice. For respect, we focused on items assessing how often the patient experienced discomfort because of a demonstrated lack of mutual respect or collaboration among team members. For role clarity, we focused on items assessing how often the patient lacked awareness of which team member to contact in the event of a question or concern.
We developed an opening statement to provide guidance to respondents completing the measure. The opening statement oriented respondents both to the scope of the health care delivery team undergoing evaluation, and the intended timeframe when responding to the questions. The opening statement read:
Receiving health care often means seeing different people, such as office staff, nurses, doctors, and other health professionals. Please think about a health issue that has led you to see different people over the last few weeks or months and answer the following questions:
Participants were adults recruited from non-clinical areas of a large tertiary hospital in the Northeastern USA. Although participants were required to be confident reading and writing in English, no other eligibility criteria were applied. We aimed to recruit a sample diverse in age with approximately equal numbers of men and women. Rather than predetermine the sample size, we aimed to continue recruitment within each interview round until saturation was reached .
One of two interviewers (RJ, SG) approached prospective participants and sought their informed consent to participate in the cognitive interviews. Participants received a written copy of the candidate items. The interviewer first asked participants to read and react to the candidate items. Next, the interviewer asked participants to describe their overall comprehension of the items by explaining the meaning of each item in their own words. The interviewer used a set of prompts to probe participants further about their understanding or interpretation of the items. Finally, participants completed a brief demographic survey. Each participant received monetary compensation for his or her time.
All interviews were recorded and summarised by the interviewer. After each round of interviews, we refined the candidate items based on participant reactions and feedback. We continued conducting interviews until saturation was reached and we had arrived at a final set of items that participants easily understood and interpreted consistently and correctly.
Again, participants were adults recruited from non-clinical areas of a large tertiary hospital in the Northeastern USA. Although participants were required to be confident reading and writing in English, no other eligibility criteria were applied. We aimed to recruit a sample of approximately 15 adults.
One interviewer (SG) approached prospective participants and sought their informed consent to participate in the pilot test. Participants were provided with a written copy of the measure developed in Phase 1 and asked to complete it with respect to a previous care experience. For the pilot administration, we used four response options for each item: Never, A little, A lot and Always. The interviewer asked each participant to offer impressions of the items based on comprehension, ease of use and salience to their experiences. We also assessed the time taken to complete the instrument. Finally, participants completed a brief demographic survey. Each participant received monetary compensation for his or her time.
The research was reviewed by the Committee for the Protection of Human Subjects at Dartmouth College and deemed exempt from further review, and institutional permission for recruitment was provided.
A total of 14 participants (6 women and 8 men) were interviewed in four rounds. Participants had a median age of 46 years (range: 19–72 years). The majority of participants did not have a university degree (n = 8). Most participants identified as White (n = 9) or Asian (n = 4), and all identified as Not Hispanic or Latino. All participants but one spoke only English at home.
Table 1 displays the original candidate items and their iterative development over time, as well as the number of individuals interviewed in each round of cognitive interviews.
The initial candidate items assessing information sharing required two changes. First, participants considered the phrase ‘more’ redundant. Second, participants felt the phase ‘work well together’ was considerably more ambiguous than ‘share information with each other’. One participant offered the thought that:
Interviewees, when asked to express the meaning of this item in their own words, consistently exhibited good understanding of the underlying idea that poor informational continuity gave rise to inefficiency and required them to spend unnecessary time or effort compensating for deficits in information transfer.
… people who share information make things go smoother and faster, which further indicates how people work well together. (male, age 65)
The item development for the domain of consistent advice focused in part on whether the use of the word ‘confused’ was better understood and more meaningful than the descriptive phrases ‘did not know what to do’ or ‘unsure what to do’. The idea of being ‘confused’ resonated more strongly. According to one participant:
We also assessed participants’ understanding of, and preferences for, the terms ‘conflicting’ versus ‘different’. Participants easily understood the phrase ‘conflicting information or advice’ and felt that its greater specificity was helpful. One participant's interpretation demonstrated the alignment in their understanding of the item:
‘confused’ strikes at the point where patients are caught being unsure of themselves and must recognize this is when they need to ask questions. (male, age 24)
By the third round, it was clear that the salient concern of patients was precisely related to being confused by conflicting advice and we therefore adopted this phrasing in the final item.
one [person] is giving me one thing and another person is giving me another. (male, age 65).
The item constructed to assess mutual respect proposed alternative phrases about the degree to which observable conflict between team members had a negative effect on patients’ sense of well-integrated care. During the second round of interviews, we added the phrase ‘lose confidence’ to the existing candidates of ‘worried’, ‘concerned’ or ‘uncomfortable’. One participant seemed to capture the overall sentiment of responses saying:
By the fourth round, it was evident that the word ‘uncomfortable’ best conveyed the sense of unease felt by participants when they witnessed conflict between team members or heard team members criticise each other. The word ‘concern’ seemed to refer to concern about other people, as suggested by this participant:
Worried – not the best word to describe losing trust or confidence because of a lack of respect among providers. (male, age 49)
Additionally, it became clear that the phrase ‘get along with’ was considerably easier to interpret than the word ‘cooperate’. As one participant said:
I'm sure people want to talk more about how they're feeling comfortable or uncomfortable rather being concerned for someone else… (female, 19)
I would think that if they get along with each other, then they wouldn't be fighting, if you know what I'm saying – like cooperating with each other, you know what I'm saying. (male, age 57)
The item constructed to assess role clarity initially focused on the issue of ‘knowing who to’ talk to about a ‘question or concern’ or ‘health issue’. During the first two sets of interviews, participants seemed to interpret this in a general sense, stating that they would seek advice from their family doctor or other generalist. This was not aligned with the intent of the item, which was to assess how often the patient lacked awareness of which team member to contact in the event of a question or concern. Using the term ‘know’ led to expectations of having specific ‘knowledge’ and was considered unrealistic by participants:
In the fourth round we confirmed that the most accurately understood phrase referred to being ‘unclear whose job it was to deal with a specific question or concern’. Being ‘unclear’ was viewed as related to:
I would leave that to the health care provider, because I wouldn't really know…, perhaps a health care provider should be the one telling you who to talk to?
(male, age 57)
The final item ‘How often were you unclear whose job it was to deal with a specific question or concern?’ was perceived positively by a participant who noted its salience:
… information not provided effectively. (female, age 53)
… just being here, and all the stuff we've been through [today], this is a great question. (male, age 46)
A total of 15 participants (7 women and 8 men) completed the pilot test. Participants had a median age of 54 years (range: 19–75 years). The majority of participants did not have a university degree (n = 10). Most participants (n = 13) spoke only English at home, with the remaining two participants speaking both English and at least one other language. The majority of participants (n = 14) identified as White, while one participant identified as Asian, and all identified as Not Hispanic or Latino.
In the pilot administration, one person had difficulty with the first item but was able to comprehend the question on her second reading. Each participant was able to complete the four items in less than one minute. All participants commented that the questions were highly relevant to patients, and that integration of care was a serious concern, especially for people with multiple or chronic conditions. Further, those interviewed said these were:
and that the questions addressed:
… problems that patients like me have experienced. (male, age 57)
… things that as a patient I have to deal with all the time. (female, age 42)
The outcome of this study is ‘IntegRATE’, a new patient-reported measure of integration in health care delivery, developed through iterative rounds of cognitive interviews with end users. IntegRATE comprises four items which map onto the unique domains of information sharing, consistent advice, mutual respect and role clarity, assessing aspects of health care delivery that are both meaningful to patients and readily modifiable. The instrument is brief and can be completed in less than one minute, maximising its suitability for use in routine performance monitoring in health care, as well as in research. The instrument is also sufficiently generic to allow completion by any individual who has recently interacted with two or more people while receiving care for a health issue, whether those people are members of an identified health care team within a single setting or members of a multi-site network of providers.
In 2010, Uijen called attention to the need to measure integration in health care delivery from the patient's perspective . Our preliminary work found that, despite a burgeoning of patient-reported measures of integration in recent years, existing instruments were typically lengthy and created for patients with specific conditions. These limitations of existing instruments led us to develop this new patient-reported measure. Strengths of our development process include a thorough review of existing approaches to assessment in the field and the identification of clearly articulated measurement domains. While these domains may not constitute a definitive conceptual model of integration in health care delivery, they comprise aspects of care frequently reported by patients to be important [7, 24), and are aligned with domains commonly assessed in other measures of integration [15, 18].
A further strength of our development process was the use of cognitive interviewing, a robust and widely advocated method for the development and refinement of survey items. At the same time, both the nature of the cognitive interview methodology adopted and the study setting resulted in considerable homogeneity in our sample of participants. Accordingly, we are cautious in generalising findings of the interpretability of the final items to the broader population. We will address this weakness during the next stages of development, which will involve testing the wider applicability and psychometric properties of the measure in more diverse populations, both under hypothetical conditions and in a broad range of clinical settings.
In all, this paper describes the first stage of work in the development and validation of IntegRATE, a patient-reported measure of integration in health care delivery. Compared with other tools , IntegRATE is brief, widely applicable and, thus, highly conducive to use in both performance monitoring and research. The next stages of development comprise the psychometric validation of the instrument and pilot implementation in routine practice.
We acknowledge all participants who contributed to this study. We also acknowledge Paul Barr, Arianna Blaine, Mary Castaldo, Emily Fletcher, Rachel Forcino, Merritt Patridge, Kenneth Rudd and Thom Walsh for providing feedback on our initial items, as well as Susan Berg and Martha Travis-Cook for their support in data collection.
Richard Byng, Professor of Primary Care Research, Centre for Clinical Trials and Population Research, Plymouth University Peninsula Schools of Medicine and Dentistry, Plymouth, UK.
Gary Oftedahl, MD, Retired, Chief Knowledge Officer, Institute for Clinical Systems Improvement, Bloomington, MN, USA.
One anonymous reviewer.
Bodenheimer, T (2008). Coordinating care — a perilous journey through the health Care system. New England Journal of Medicine Mar 6 2008358(10): 1064–71.
Ham, C (2010). The ten characteristics of the high-performing chronic care system. Health Economics, Policy and Law Jan 20105(Pt 1): 71–90.
Kripalani, S, LeFevre, F, Phillips, CO, Williams, MV, Basaviah, P and Baker, DW (2007). Deficits in communication and information transfer between hospital-based and primary care physicians: implications for patient safety and continuity of care. JAMA Feb 28 2007297(8): 831–41.
Uijen, AA, Schers, HJ, Schene, AH, Schellevis, FG, Lucassen, P and van den Bosch, WJHM (2013). Experienced continuity of care in patients at risk for depression in primary care. The European Journal of General Practice Sep 13 201320: 161–6.
Uijen, AA, Bosch, M, van den Bosch, WJHM, Bor, H, Wensing, M and Schers, HJ (2012). Heart failure patients' experiences with continuity of care and its relation to medication adherence: a cross-sectional study. BMC Family Practice Jan 201213(1): 86.
Mastellos, N, Gunn, L, Harris, M, Majeed, A, Car, J and Pappas, Y (2014). Assessing patients' experience of integrated care: a survey of patient views in the North West London Integrated Care Pilot. International Journal of Integrated Care Apr 201414: e015.URN:NBN:NL:UI:10-1-114782
McKinnon, LC, Prosser, SJ and Miller, YD (2014). What women want: qualitative analysis of consumer evaluations of maternity care in Queensland, Australia. BMC Pregnancy and Childbirth Oct 26 201414(1): 366.
Fortin, M, Bravo, G, Hudon, C, Vanasse, A and Lapointe, L (2005). Prevalence of multimorbidity among adults seen in family practice. Annals of Family Medicine Jan 1 20053(3): 223–8.
Sun, X, Tang, W, Ye, T, Zhang, Y, Wen, B and Zhang, L (2014). Integrated care: a comprehensive bibliometric analysis and literature review. International Journal of Integrated Care Apr 201414: e017.URN:NBN:NL:UI:10-1-114784
Kodner, DL and Spreeuwenberg, C (2002). Integrated care: meaning, logic, applications, and implications – a discussion paper. International Journal of Integrated Care Jan 20022: e12.URN:NBN:NL:UI:10-1-100309
Evans, JM and Baker, GR (2012). Shared mental models of integrated care: aligning multiple stakeholder perspectives. Journal of Health Organisation and Management Jan 201226(6): 713–36.
Valentijn, PP, Schepman, SM, Opheij, W and Bruijnzeels, MA (2013). Understanding integrated care: a comprehensive conceptual framework based on the integrative functions of primary care. International Journal of Integrated Care 13: e010.URN:NBN:NL:UI:10-1-114415
Singer, SJ, Burgers, J, Friedberg, M, Rosenthal, MB, Leape, L and Schneider, E (2011). Defining and measuring integrated patient care: promoting the next frontier in health care delivery. Medical Care Research and Review Feb 201168(1): 112–27.
Strandberg-Larsen, M and Krasnik, A (2009). Measurement of integrated healthcare delivery: a systematic review of methods and future research directions. International Journal of Integrated Care Jan 20099: e01.URN:NBN:NL:UI:10-1-100547
Uijen, AA Heinst, CW Schellevis, FG van den Bosch, WJHM van de Laar, FA Terwee, CB et al. (2012). Measurement properties of questionnaires measuring continuity of care: a systematic review. PLoS One Jan 7(7): e42256.
Aller, MB Vargas, I Garcia, I Coderch, J Colomés, L Llopart, JR et al. (2013). A tool for assessing continuity of care across care levels: an extended psychometric validation of the CCAENA questionnaire. International Journal of Integrated Care Dec 2 201313(4): e050.URN:NBN:NL:UI:10-1-114758
Bentler, SE, Morgan, RO, Virnig, BA and Wolinsky, FD (2014). Evaluation of a patient-reported continuity of care model for older adults. Quality of Life Research [Internet] Feb 201423(1): 185–93. [cited 2014 Mar 27]. Available from: http://www.ncbi.nlm.nih.gov/pubmed/23868458
Haggerty, JL, Roberge, D, Freeman, GK, Beaulieu, C and Bréton, M (2012). Validation of a generic measure of continuity of care: when patients encounter several clinicians. Annals of Family Medicine [Internet]. American Academy of Family Physicians Jan 1 201210(5): 443–51. [cited 2014 Mar 28]. Available from: http://www.annfammed.org/content/10/5/443.full
Bowling, A (2005). Just one question: If one question works, why ask several?. Journal of Epidemiology and Community Health [Internet] May 1 200559(5): 342–5. [cited 2013 Apr 14]. Available from: http://jech.bmj.com/content/59/5/342.full
Willis, G (2005). Cognitive interviewing: a tool for improving questionnaire design. Thousand Oaks, CA: Sage.
Haggerty, JL, Roberge, D, Freeman, GK and Beaulieu, C (2013). Experienced continuity of care when patients see multiple clinicians: a qualitative metasummary. Annals of Family Medicine [Internet] Jan 1 201311(3): 262–71. [cited 2014 Jul 15]. Available from: http://www.annfammed.org/content/11/3/262
King, J, Gibbons, E, Graham, C and Walsh, J (2013). Developing measures of people's self-reported experiences of integrated care. Oxford: Picker Institute Europe and University of Oxford.
Naithani, S, Gulliford, M and Morgan, M (2006). Patients' perceptions and experiences of “continuity of care” in diabetes. Health Expectations Jun 9(2): 118–29.
Walker, KO, Labat, A, Choi, J, Schmittdiel, J, Stewart, AL and Grumbach, K (2013). Patient perceptions of integrated care: confused by the term, clear on the concept. International Journal of Integrated Care 13: e004.URN:NBN:NL:UI:10-1-114281
Uijen, AA, Schers, HJ and van Weel, C (2013). Continuity of care preferably measured from the patients' perspective. Journal of Clinical Epidemiology Sep 201363(9): 998–9.