Start Submission Become a Reviewer

Reading: Evaluation is Key: Providing Appropriate Evaluation Measures for Participatory and User-Cent...


A- A+
Alt. Display

Research & theory

Evaluation is Key: Providing Appropriate Evaluation Measures for Participatory and User-Centred Design Processes of Healthcare IT


Lorenz Harst ,

Center for Evidence-based Healthcare, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, DE
X close

Bastian Wollschlaeger,

Chair of Technical Information Systems, Technische Universität Dresden, DE
X close

Jule Birnstein,

Zittau/Görlitz University of Applied Sciences, DE
X close

Tina Fuchs,

Chair of Technical Information Systems, Technische Universität Dresden, DE
X close

Patrick Timpel

Center for Evidence-based Healthcare, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden; Prevention and Care of Diabetes, Department of Medicine III, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, DE
X close


Introduction: The increasing availability of healthcare IT has the potential to improve the integration of health services. Existing projects developing healthcare IT mostly disregard the potential and importance of incorporating user feedback and proper evaluation measures to gain user feedback throughout the development process. We therefore provide methodological guidance for evaluation in a stepwise user-centred design process.

Methods: Based on a literature review we propose adequate methods for data collection in each phase of participatory and user-centred healthcare IT development. In order to provide an orientation within the plethora of development processes used in practice, we consolidate a generic blueprint process from the literature review. The applicability of our methodological guidance is shown in three diverse use cases from the field of integrated care.

Results: From 14 literature items, we identified common evaluation methods including, among others, interviews, focus groups, and surveys. These methods can be associated to six typical development phases that could be derived from research: State of the Art Research, Requirement Analysis, Conceptual Prototype, Preliminary Prototype, Full Prototype, Full Application. The use cases demonstrate the value of qualitative methods and mixed methods designs.

Discussion: Our methodological guidance has proven applicable for designing healthcare IT solutions from scratch – both for patient and professional settings – and to develop a platform for combining existing component-based solutions. In integrated care settings, where a wide range of stakeholders with multiple needs exist, we thus provide methodological guidance on how to involve users in the development process.

Conclusion: Our stepwise methodological guidance helps to design and properly evaluate healthcare IT solutions, which meet the user needs and requirements, for integrated care settings.

How to Cite: Harst L, Wollschlaeger B, Birnstein J, Fuchs T, Timpel P. Evaluation is Key: Providing Appropriate Evaluation Measures for Participatory and User-Centred Design Processes of Healthcare IT. International Journal of Integrated Care. 2021;21(2):24. DOI:
  Published on 21 Jun 2021
 Accepted on 16 Mar 2021            Submitted on 30 Apr 2020


Patients with chronic conditions are in need of medical care delivered by multiple stakeholders [1]. Apart from that, they often experience a variety of competing needs, which are not limited to morbidity-related aspects, but include, e. g., the need for a social support network [2]. These needs, prioritized differently from one individual patient to another, often remain unmet, partially due to a lack of funding [3]. In response to this, targeted chronic care, e.g. individualised glycaemic targets and decision-making for patients with diabetes, is seen as a key to address these unmet needs [4].

Due to this demand for individualised care measures, chronically ill patients are the ideal target population for digital health devices, especially those that fall under the definition of telemedicine [5]. These devices can provide continuous monitoring of clinical and behavioral parameters, such as diet and physical activity [6], and, above all, individualised health care provider feedback based on the collected values [7]. As the emergence of mHealth applications allows for tailoring digital health interventions to the needs of specific patient groups [7], applying development processes that allow for continuous user participation becomes even more pressing. While, lately, the concept of patients as consumers of digital health applications has gained prominence [8], patients still differ from other technology users in several key characteristic. As the example of diabetes self-management shows, patients need to continuously monitor their blood glucose levels as well as dietary and physical activity behaviors [9]. Such, devices that aid this process have to be used continuously as well in order to be effective [10]. Discontinued use due to loss of motivation, as it can be the case with any digital device [11], therefore poses a threat to therapeutic outcomes. Furthermore, potential users, i. e. patients with chronic diseases, are mostly older, less interested in technology in general and therefore less skilled in using novel devices, and, maybe above all, used to and reliant on the personal connection to their doctor [12, 13]. Fitting a digital device to the user needs becomes even more complicated when both patients and health care providers need to use the device. For instance, in diabetes care, research on patients’ and health care providers’ needs and expectations indicates significantly different priorities [3]. Furthermore, privacy concerns, which are often disregarded by users when it comes to sharing holiday pictures, amount to a usage barrier when personal health data are to be shared [14].

A recent synthesis of barriers in telemedicine implementation processes indicates that the majority of these barriers arise from a lack of theoretical underpinnings for individual end-users’ acceptance [15]. Theories of technology acceptance, which include characteristics of both users and the technology as predictors of acceptance, should be used to guide the design process [16]. Thus, the acceptance of any healthcare IT application by patients and their direct social environment, as well as by health care providers, can be ensured early within the design process. A non-consideration of acceptance and users’ preferences during the development of a health IT application may ultimately lead to non-effective applications due to low acceptance and usability rates [17]. Coming from IT development, therefore, the groundwork of Norman and Draper for user-centred design [18] is also demanded by the ISO norm 9241-210 for interactive systems [19]. A focus on perspectives of both providers and patients in the design process [20] is in line with recent calls to shift from process evaluations of integrated care solutions to a more participatory form of implementation [21] and evaluation [22], commonly called participatory design [23].

Before being applied in standard care, healthcare IT and digital devices in healthcare have to provide evidence on safety and/or the effectiveness regarding clinical or health-related outcomes. However, due to the still overarching dominance of IT experts and institutions in the development processes of such devices, both the participation of end-users (patients and providers alike) as well as evidence-based data collection using adequate methods is still lacking [8, 24]. To this date, there is a number of prototypical processes for the user-centred design of specific health care IT applications, such as teleconsultation systems [25], web-based health information databases [26] and consumer mobile health applications [27]. For integrated care systems in a community setting, Wodchis et al. propose a standardized implementation framework [28], which, however, is not based on existing concepts and does not provide methods to support the intended user and provider participation.

Based on a literature review, we provide methodological guidance indicating how to collect data e. g. on users’ needs, acceptance and priorities within the relevant development phases of a generic blueprint for user-centred and participatory design procedures, which can be applied by developers of future healthcare IT applications. Based on three use cases, we then demonstrate the merits of applying some of the proposed methods in the development process of different digital health applications in integrated care settings.


Since in practice a variety of models for user-centred and/or participatory application development are used, such as the ones proposed by Esser & Goossens [25], Arsand & Demiris [8] and Sutcliffe & colleagues [31], and no standard development model can be identified, we first consolidated a generic blueprint for these processes. The blueprint is not intended to serve as a suggestion on how to develop healthcare IT applications, but rather shall be used by developers as a common ground for choosing appropriate evaluation methods during the development process.

To this end, we conducted a literature review [29] to derive a generic blueprint for user-centred and participatory design processes, which then was used as a reference for when to apply which evaluation methods.

The review of models for healthcare IT development and evaluation was conducted within PubMed and Google scholar. The former was chosen as it is likely to include research on adequate methods for user-centred and participatory design processes. The latter was chosen to complement the search. Technical databases such as IEEEXplore were explicitly disregarded as they rather deal with research on technical feasibility and focus less on evaluation aspects.

Search terms included synonyms for user-centred design, user acceptance and the terms “framework” or “model”. Only research proposing and/or applying a complete user-centred design process was included. No limitations for publication types, domains (prevention, primary or inpatient care) or disease types addressed by the healthcare IT application in question were defined. The same applies to publication dates.

Based on the results, a generic blueprint for the user-centred and participatory design of healthcare IT applications was derived. First, the design and development steps suggested in more than one of the included references were categorized and sorted chronologically in an inductive process derived from Mayring’s qualitative content analysis. This method is based on rearranging text material in order to uncover underlying patterns [30]. In order to validate the applicability of our thus derived generic blueprint, we aligned it with existing prominent models of user-centred and/or participatory design of health care IT, such as the ones mentioned above.

The evaluation methods suggested in the included literature were matched with the phases of the generic blueprint based on suggestions made in the identified literature. The literature review, as well as the consolidation of the blueprint and the mapping of the methods, were carried out independently by two researchers in order to minimize bias.

Three use cases, one for a mHealth application for diabetes self-management, one for a digitally supported discharge management system coordinating patients’ transition and post-hospital care, and one for an automated design method for Ambient Assisted Living (AAL) solutions, will illustrate the applicability of the methods suggested in each step. Short paragraphs on lessons learned are provided to critically reflect on the applied methods and their role in the use cases.


First, a generic blueprint for user-centred and participatory design processes of healthcare IT applications is presented as a means to provide structure for associating the evaluation methods. In a second step, appropriate evaluation methods for each phase are suggested. In a final step, three different use cases are presented and discussed illustrating the applicability of the methodological guidance.

The generic blueprint for user-centred and participatory design processes

Derived from 14 research items identified in the review process, the generic blueprint for user-centred and participatory design processes consists of six-phases and will be used as a canvas for the allocation of adequate methods. The allocation of each evaluation method to the phases is depicted in appendix 1. As a result of the review and qualitative alignment process, Figure 1 depicts the different phases, including a short explanation of the tasks required in each phase.

Figure 1 

The phases of the generic blueprint for user-centred and participatory design processes.

Phase 0 covers relevant state of the art analyses, including theoretical basis, which should be considered during all consecutive steps. This state of the art research may comprise systematic or rather narrative review (e. g. desk research) methods [29] to identify guidelines, theories or other relevant projects to be used as a starting point. Phase 1, requirement analysis, is the pre-prototype phase, where user requirements, i. e. medical or professional needs and expectations towards a potential application, are gathered, preferably within the context the application will be used in. “Context” needs to be differentiated between the professional setting of health care providers and the social environment of patients.

The next phases constitute the prototype stages: In Phase 2, the results from Phases 0 and 1 are used to develop a conceptual prototype of the application, be it an actually usable prototype or just templates or sketches that depict the design (or main concept) of the application, including its layout. In Phase 3, the prototype is refined based on a first usability evaluation in controlled environments. Especially the user interaction with the prototype needs to be monitored and analysed leading to a preliminary prototype. This process is repeated in Phase 4 using an evaluation under real-life conditions with a full-scale prototype of the application, so that final changes can be made to improve ease of use and, therefore, user acceptance.

This concludes the prototype stages, so that finally, a full-scale application can be implemented and subjected to long-term evaluation in Phase 5. It is important to note that the process is iterative; meaning that in each phase the researchers and/or developers may take a step back to the previous phase if s/he thinks that important insights are missing.

The described prototypical participatory user-centred design process uses the background variables (input) defined by Esser & Goossens [25], including individual, organisational and technical contexts, to account for the complex interrelations of requirements healthcare IT has to deal with throughout the whole development process. Using these patient and context requirements to develop a patient-centred application that will be accepted by the patient is also supported by Arsand & Demiris [8]. The authors further call for context- and user-group-relevant methods being applied to support the iterative development of the prototype [8]. Finally, Sutcliffe & colleagues [31] argue that the evaluation of prototypes further helps to specify the volatile and complex requirements of the healthcare sector.

Methods used for data collection in each Phase

Established evaluation methods are now presented with respect to their characteristics as a foundation for associating the appropriate methods to each phase. Table 1 depicts the relation between the different process phases and the respective applicable methods.

Table 1

Applicability of methods for the different phases.


0: State of the art research x

1: Requirement analysis x x x x x x x x x x x

2: Conceptual prototype x x x x x x x

3: Preliminary prototype x x x x x x

4: Full prototype x x x x

5: Full application x x x x

Qualitative methods (e. g. semi-structured interviews and focus groups) are useful for gathering in-depth insights into user needs and expectations in Phase 1 [8]. They can be also used in the field, i. e. the natural environment of patients or the professional environment of health care providers in Phases 3 and 4 [32]. Such, they are ideal – albeit time-consuming – for understanding the context of use. Alternatively, quantitative surveys can be used for gathering the same information [33], yet without a deeper understanding of the users’ specific needs and expectations.

Existing personas may inform first needs for adaptation [34]. For developing life-like and application-specific personas, qualitative methods could be more apt [35], while user profiles can also be derived from cluster analysis based on survey data [36].

For the first prototype, testing it with reference to realistic use cases is essential. Discussing these use cases in focus groups or interviews can help to overcome the first hurdles that could hamper acceptance [37]. While actual working prototypes help to better simulate a usage scenario, paper prototypes or sketches are less cost-intensive alternatives [38]. In addition, changes in fully developed prototypes or systems are more labour-intensive as they might entail system-wide adaptions [39].

However, task-driven think-alouds [40] or cognitive walkthroughs [41] need to be conducted with a full-scale prototype in Phases 3 or 4. The same is true for quantitative system usability surveys as a more standardized alternative. Here, the IBM system usability scale, which measures satisfaction with the effort it took to achieve a task [42], is an apt choice. However, the more complex the system, the more usability issues can arise in the prototype phases. In order to prioritize challenges for the usability and the changes in the system they entail, a severity ranking can be used, ranging from “not a problem” (if a problem is only stated by a single user) to “usability catastrophe” (a problem that definitely has to be fixed before release of the application) [43].

Use Cases demonstrating the applicability of the methodological guidance

Finally, in a third step, the applicability of the methodological guidance suggested is illustrated in three use cases highlighting the potential impact of working with the allocated methods in each phase of our generic blueprint process.

Use Case 1: Development and Testing of a Diabetes Self-Management App

The methodological guidance was applied in a study to develop the first diabetes self-management (DSM) app in Rwanda [44], called Kir’App. Diabetes self-management highly depends on feedback from various health care providers, e. g. primary physicians, diabetes educators or dietitians [9]. In this study, a mixed methods approach for the user-centred design of a diabetes self-management app was chosen. First, evidence for effectiveness of theory-based behaviour change interventions was gathered (Phase 0), suggesting that applications including features for patient-provider interaction and educative content are especially promising in terms of reductions in blood glucose levels [45].

Afterwards, 21 Rwandan patients with diabetes – as potential users of the envisaged app – were asked in semi-structured face-to-face interviews for their expectations towards a self-management app. The patients were contacted personally at the offices of the Rwandan diabetes association. Results of a qualitative content analysis of the interview transcripts indicate a need for diabetes-related knowledge, especially on how to conduct the daily life (nutrition and physical activity), monitoring of blood glucose values and reminders for medication intake as well as physical activity, and social support features such as means to contact other diabetics. In parallel, an analysis of local care structures and care practices, including diabetes-related health literacy, was conducted to explore unmet needs as relevant determinants of the context of use (Phase 1). As a result, a reliance on traditional medicine as a supplement to the existing healthcare systems in place was uncovered, as well as a lack of structured diabetes care programs [44].

Based on these qualitative insights, requirements for the mobile health app were categorized and a first prototype was developed, which contained the ten functionalities depicted in Figure 2, ranging from information provision over a personal diabetes logbook to a reminder function (Phase 2).

Figure 2 

Functionalities of Kir’App.

In a next step, this prototype was made available to the same patients who were initially asked for their expectations. Fourteen of these chose to participate. This time, their experiences were collected after they used the app for three consecutive months. Again, semi-structured interviews were applied to assess whether the first prototype of the app met their overall needs and expectations. The categories gained in Phase 1 could all be identified again in the interview transcripts, showing that the prototype was in line with user expectations. Usability issues occurred e. g. when setting the alarm for medication reminders and could be sorted out afterwards. All in all, participants stated using the app increased their diabetes-related knowledge and aided in diabetes self-management [46]. In addition, a usability test based on the IBM system usability scale was conducted [42] (Phase 3). Preliminary results indicate that participants did not need exceptionally long to perform certain tasks and were satisfied both with the effort it took to achieve these tasks and with the overall usability of the app.

In a next step, a randomized controlled trial (RCT) will be conducted to evaluate the effectiveness of the app in terms of clinical outcomes (blood sugar levels; HbA1c) in Rwandan patients with diabetes (Phase 4). A focus group being carried out after or nested within the RCT will help to uncover specific functionalities of the application serving as drivers and barriers of sustained use, while also exploring factors affecting recruitment and implementation [47, 48].

Evaluations of effectiveness will then be accompanied by a continuous assessment of user satisfaction, e. g. via an in-app questionnaire (Phase 5) [49]. The app is available in the Google Play Store and used by mainly Rwandan diabetics.

Lessons learned: For collecting user expectations towards a potential diabetes self-management application from diabetes patients, semi-structured interviews have proven especially valid because they provided a setting of confidentiality, so that knowledge gaps about the disease as well as difficulties with adhering to treatment regimes in everyday life could be uttered freely. Moreover, semi-structured interviews proved beneficial because they allowed for collecting heterogeneous – sometimes-conflicting – perspectives from a sample consisting of patients with heterogeneous levels of health literacy and experience with digital technologies. The literature research warranted in Phase 0 was essential to understand the cultural background of the participants in the interview sessions.

Use Case 2: A discharge management online platform

The number of beds per capita as well as lengths of hospital stays are continuously reduced in most European countries [50]. The latter is regarded as a measure of efficiency [51]. Although shorter stays are intended to reduce costs by moving patient care from the highly service intensive and costly inpatient care setting to ambulatory care, this requires intensive aftercare coordination during the short hospital stays [52]. Based on the methodological guidance, a discharge management online platform was developed by a German statutory insurance company in collaboration with a university clinic to facilitate this organizational process.

In Phase 0, state of the art research indicated that discharges from hospitals are prone to care fragmentation and losses of patient information and may lead to post-discharge readmissions [53]. The competing medical and social demands of chronic care patients urge for a structured, timely and medically acceptable coordination of the hospital discharge and care transition process without putting the patient at risk [53]. Digital discharge management strategies need to support nurses and other personnel in coordinating relevant post-acute services and remedies as well as to apply for funding of these services with the health insurers [54]. However, no digital device of this configuration exists to date.

Therefore, semi-structured interviews with nurses and other personnel responsible for the discharge management were conducted to determine possible aspects of the application process to be improved by an online platform (Phase 1). The relevant personnel was recruited via E-Mail and in recurring meetings by the head of the hospital social service units. The results show that the personnel involved wished for an online application that improves knowledge of a patient’s situation (concerning e. g. her/his residential situation as well as assistance systems already in place) early after admission. The latter would save time in the discharge process, guaranteeing earlier discharge. Furthermore, the personnel interviewed whished for an uplink to the existing hospital information system, paperless processes, including an option for digital signatures, and an intuitive user interface. Above all, they were adamantly opposed to a system which would only be applicable for interacting with one instead of all insurance agencies, as this would lead to extra work.

The first software prototype (Phase 2) was subject to a technical testing before nurses, case managers and social service personnel at the university clinic were trained in two sessions in using the software interface. The training sessions were observed openly by two researchers (field study). Questions arising in the sessions were collected and used to re-adjust the interface (Phase 3). For example, during the training sessions, the participants expressed a need to be able to proceed through the systems without knowing the exact ICD-10 diagnosis code, and to upload necessary documents days after first dealing with a patient within the system.

After a first live testing phase (Phase 3) of four weeks in orthopaedics and trauma surgery wards, technical issues had to be resolved before the online platform was used in almost all wards equipped with patient beds.

Although focus groups were preferred to engage discussion among the users of the online platform for the next evaluation step (Phase 4), restrictions due to COVID-19 caused a switch to an online semi-standardised survey. Focal topics of the online survey included an assessment of users (including technical affinity and work experience in years) and usage of the online platform measured in days. Afterwards, participants were asked to list problems in their line of work, focussing on the period before using the online platform. The subsequent questions focussed on menu navigation and general design of the app. The integration of the system into everyday clinical practice in the wards and its usefulness when dealing with complex cases, e. g. patients with multiple chronic conditions, was subject of further questions. The system usability scale, a standardised 10-item questionnaire [55], was used to measure user satisfaction and ease of use. Usability testing also included several use cases depicted in screenshots of the interface as part of the survey.

Finally, participants were asked to describe their wishes for a “perfect online platform in discharge management”. Consequently, the mapping of a) problems in the everyday line of work of discharge managers, b) strengths and c) weaknesses of the online platform as well as d) reflections on a “perfect online platform” from a user perspective are expected to pave the way for additional functions and updated versions of the online platform, which is currently in re-development mode.

After finalising the full application, economic analyses will be conducted to evaluate changes in terms of length of stay or 30-day readmission rates. These analyses will be accompanied by continuous assessments of perceived usefulness by the nurses.

Lessons learned: The initial semi-structured interviews proved to be a time-saving way to gather initial expectations towards the discharge management system. However, they also showed that participants had difficulties envisioning such a system. Therefore, the field study during the training session with the first functional prototype yielded more practical feedback on how to improve the system. The same is true for the semi-structured online questionnaire, which also allowed for linking quantitative assessments of usability with qualitative insights on the system as a whole, showing that missing functionalities outweigh an overall good usability. It also allowed to gain insights from case managers working in different medical disciplines and into corresponding requirements towards the new system.

Use Case 3: Assessment of an Automated Design Process for customized Health Smart Homes

In contrast to individual digital health applications, Ambient Assisted Living (AAL) or Health Smart Home (HSH) systems [56] are complex multi-component-based systems that are composed of already existing applications and products [57]. Integrated care in the context of AAL therefore refers to integrating a set of individual monitoring and assistance tools to aid chronically ill patients [58]. In this case, customization in a user-centred design process is achieved by an appropriate selection and integration of components-off-the-shelf, without the need for development or alterations of the individual products themselves. The appropriateness of components is mainly determined with regard to the users’ needs, which are motivated by the users’ medical conditions; however, general user preferences can also be taken into account [59].

Since it offers an orthogonal approach of customization, this domain is suitable to demonstrate the broad applicability of the presented methodological guidance. Therefore, this use case will discuss the application of the proposed methodological guidance in developing an automated AAL suggestion system. This AAL suggestion system has been proposed on a theoretical level by Wollschlaeger and Kabitzsch [60], and facilitates the design and tailoring process for AAL solutions during patient consultation. AAL consultation is a two-tier process, consisting of the actual consultation and subsequent follow-up evaluations. Addressing the consultation activities, the automated AAL suggestion system is intended to be used by AAL counsellors in an interview-style setting with a patient, yielding possible and need-based system candidates. A prototype of the suggestion system has been implemented as a web-based application (cf. Figure 3) and facilitates a participatory approach to composition and tailoring of component-based systems. Thus, the actual artefact that is being developed using a participatory design approach is the customized and need-based Health Smart Home.

Figure 3 

Start screen of the web application for suggesting need-based assistance systems inviting users to find the right solution for their Health Smart Home. In the lower part, information on health samrt homes can be procurred and different options for interacting with the system are offered.

As a consequence, the guidance was not primarily used for developing the AAL suggestion system itself, but it rather provided orientation on how the AAL suggestion system should facilitate the consultation process.

The web-based application starts with an analysis of the user requirements (Phase 1). These requirements are subsequently used to determine appropriate abstract assistance concepts, tailored to the respective user (Phase 2). Finally, suitable assistance components implementing the required assistance concepts are selected from a database and suggested to the AAL counsellor and the patient. At the end of the consultation, a customized AAL system can be selected based on these suggestions and installed at the patient’s home (Phase 3).

The different steps provided by the described AAL suggestion system are in accordance with the blueprint and the provided methodological guidance. On the one hand, this provides guidance and justification for the early Phases 1 to 3. On the other hand, suitable evaluation methods that are valuable for designing the organisational framework for the suggestion system (Phases 3 to 5) can be identified from the proposed method set.

More specifically, in Phase 1, the user requirements are analysed based on a standardized questionnaire, which has been developed according to personas, user profiles of patients and insights of health care providers (cf. Figure 4). Both the use of pre-defined profiles, including dementia, diabetes, geriatrics, or stroke, and a custom configuration of user requirements is supported by the AAL suggestion system.

Figure 4 

Screen for starting with a pre-defined patient profile. Profiles to be chosen are dementia, diabtes geriatrics/multi-morbidity, stroke, and free selection of symptoms.

The concept development of Phase 2 is realized by suggesting appropriate assistance concepts on an abstract functional level (cf. Figure 5), thereby sketching the intended applications and providing a rough prototype. In the presented example, the suggested assistance concepts include automated equipment deactivation, detection of fire and smoke, reminder & calendar functionalities, and medication support.

Figure 5 

The suggested assistance concepts of the consultation web app are ranked according to their usefulness for the chosen profile.

The subsequent suggestion of specific assistance products provides input for the development of a specific prototype in Phase 3 (cf. Figure 6). For each assistance concept identified in the previous step, suitable assistance products are proposed. As can be seen from Figure 6, the functionality “activity monitoring” can be provided by the step counters connected to mobile apps such as my sugar, which is an app for diabetes prevention [61], or FAME, an app developed for patients with early-stage dementia [62].

Figure 6 

At the end, assistance products are proposed for each suggested assistance concept.

As a consequence, the AAL suggestion system is a tool offering support for the first steps of a user-centred design process.

The further evaluation actions of Phases 3 to 5 address the organisational context of the suggestion system. Thus, the method set proposed by the methodological guidance introduced in this article provides guidance on how to design evaluation steps and follow-up evaluations. It can be inferred from the description of Phase 3 that shortly after installing the assistance components in the user’s home, a follow-up evaluation regarding usability is warranted to make sure the user can operate the components as intended. Based on the method set for Phase 3, interviews, think-aloud studies, and usability surveys as described in use case 1 could be appropriate tools for this evaluation. After this first tutorial phase, the user should be familiar with the new components and their use. It is now possible to assess the efficacy and user acceptance in real-life conditions using field observations and interviews (Phase 4). Finally, the long-term effects on the patient’s quality of life or her/his coping with the disease, as well as possible need for adaptations can be evaluated using semi-structured interviews or quantitative surveys with the patient, the AAL counsellor, and the patient’s general practitioner (Phase 5). At any given time, a new iteration of the consultation process can be started, as suggested by the blueprint in section 5.1.

A prototype of the suggestion system was implemented and evaluated by both case managers and patients. Further development and introducing support for the last phases of the design process as well as the follow-up evaluations is a currently ongoing activity in order to enhance the applicability of the suggestion system [60].

Lessons learned: In this use case, the developed AAL suggestion system itself is used as a facilitator for a user-centred and participatory selection and composition of assistance technology for patients requiring technical assistance in their homes. During the development of the suggestion system, the evaluation guide proved beneficial for selecting possible interaction patterns with the users. Thus, a questionnaire setting combined with personas and exemplary user profiles could be suggested as suitable for analysing the patients’ requirements. Initial test users of the prototype of the AAL suggestion system (medical experts such as case managers and practitioners, but also potential patients and their relatives) confirmed this setting to be a viable approach for the analysis of requirements. Instead of presenting the suggested assistance components straight after this analysis, we consulted the guide and chose to present intermediate results (in terms of neutral assistance functionalities) as a kind of application sketch. The feedback provided by the initial test users indicates that this step-by-step approach facilitates a deeper understanding of the proposed assistance components by the patients.

As we are fully aware that the decision support offered by the AAL suggestion system calls for follow-up evaluations, we identified the proposed evaluation methods of Phases 3 to 5 as helpful input when designing the organisational context the suggestion system should be used in. Yet, further practical application of the suggestion system is required in order to assess how beneficial the contributions of the evaluation guide for the organisational context design actually was.


Summary of results

Using the phases of a generic blueprint for user-centred and participatory design processes of health care IT as a starting point, we provided adequate methodological guidance as a scientific foundation for the participatory development process of healthcare IT applications. This, in turn, limits the risk that the success of applications is not hampered by its non-adoption by the prospective users [2]. Our three use cases demonstrate the applicability of our proposed methods and the phases to which they are assigned for developing a self-management app from scratch and without previous knowledge of the user group, but also for building a discharge management platform in a hospital setting and for combining existing AAL and HSH solutions.

The application of the blueprint, enhanced with established scientific methods of evaluation, enables health care IT developers to address the complex requirements of healthcare IT mentioned in the beginning. Following the methodological guidance for each generic phase helps capturing the interrelations between heterogeneous patient characteristics (due to age, health literacy, individual health care preferences, previous experiences with technology, and beliefs of self-efficacy [16]), the communication between and cooperation of various stakeholders, and health care system requirements relevant to reimbursement of health services. As such, the generic nature of our methodological guidance may support especially healthcare IT developers with less experience in evidence-based evaluations of healthcare services to adapt and apply our recommendations to heterogeneous use cases and settings in a step-wise manner.

Comparison to prior research

A recent analysis revealed a wide range of definitions and applications of relevant terms like acceptability, acceptance and adoption [63]. Our generic blueprint for user-centred and participatory design processes, being accompanied by established methods for evaluation, may also contribute to an improved level of standardisation. In general, our process is in line with existing practices for user-centred design as they are made publically available, e. g. by the platform “” [64]. It also corresponds to the principles of Cooperative Design, as we aim for the participation of all relevant stakeholders in the design process [65]. However, our approach necessarily goes beyond the existing ones: With respect to digital health applications used in the integrated cares setting, the patient-provider interaction as well as the interaction between providers needs to be in the focus of all six steps. Theoretical frameworks for the integrated care of the chronically ill, such as the Chronic Care Model [66], should drive the development of the first prototype [67]. The needs assessment should also cover the requirements of patient-provider- and provider-provider communication [68]. As for the full-scale implementation and continuous evaluation of healthcare IT applications in an integrated care setting, changes in the interaction, in particularly the shared decision making process, need to be monitored [69]. However, patient-provider interaction remains unaccounted for in several implementation processes for integrated care systems, e. g. for the Central Coast Integrated Care Program [70].

In addition, a clear assignment of adequate methods to each consolidated step has not been provided so far. Consequently, analyses of established European integrated care programs targeting patients with multi-morbid conditions show that these are insufficiently equipped with adequate methods to support patient involvement [71]. Our research contributes to closing this gap and thereby helps decreasing the risk of methodologically flawed evaluation studies [72], where, e. g., Patient-Reported Experience Measures are disregarded [73].

As the development of user-centred interventions and application systems is a complex phenomenon, we take into account not only the individual him-/herself, but also his/her context of use, which is represented, e. g., by his/her conduct of daily life and disease coping mechanisms [74]. Whether initial expectations have been met needs to be considered before roll-out of any technological innovation [19]. Therefore, our several iterative steps (Phases 2 to 5) are incremental and in line with the PDCA cycle [75] as well as the established key elements of integrated care [76]. The idea of conceptualising product development levels and evaluating the maturity of the product accordingly is the focus of the established technology readiness assessment [77], as well as an essential part of Roger’s Diffusion of Innovations theory [78]. Our proposal puts a stronger emphasis on individual and contextual characteristics [79] and how they can be captured best with scientific evaluation methods. As such, it is likely that an integration of our methodological guidance into existing procedures, like models frequently applied in software development projects (e. g. spiral models [80], V-model or agile approaches like Scrum [81]; for a conclusive overview see [82]) may have an additional benefit, especially since those often lack an evidence base. As we aim to provide methodologies for co-creating healthcare IT applications, they fit very well into existing living lab approaches [83]. An early consideration of the end user, i. e. his/her needs and expectations, is one of the methodological requirements according to the newly developed NICE framework for digital health evaluation. Levels of evidence stratified by functional classifications as introduced by the NICE framework may further guide the evaluation and certification process in all of our use cases [84].

Our approach puts an emphasis on methods that allow user participation throughout the whole process. Such, it promotes a shift from user-centred to participatory design [23].

From a methodological standpoint, we provide methods from both the qualitative and quantitative realm, which is in line with the notion that mixed methods might be best suited to capture the complex requirements of implementation of novel techniques in health care [85]. While the methods we propose are not new themselves, we demonstrate how they can be adapted and combined to serve the complex requirements posed by the health care sector (see introduction).


Although our proposal is based on recent evidence in the field of healthcare IT, our approach has limitations. First, no criteria-driven inclusion process was carried out to identify and include relevant evidence. As we did not aim at developing an all new user-centred design framework, but rather aimed at collecting methods to be applied to ensure user participation in each step of a generic blueprint, we found a literature review [29] to be sufficient. However, a broader set of databases, such as, Embase Web of Science or Scopus, could have yielded more results. As such, we may have missed relevant manuscripts and the identified methods corresponding to the phases may not be exhaustive. One example is the consideration of adequate study designs in Phases 4 and 5. It is recommendable to choose study designs that correspond to the types and functionalities of the innovation and simultaneously consider the maturity and efficacy of the prototypes.

As a common limitation of qualitative studies, the individual backgrounds and subconscious expectations of the authors may have also had an influence on the consolidation of phases and the mapping of methods. As such, the suitability of the proposed methodological guidance for providing orientation in selecting adequate methods during the development process requires further validation, especially concerning its Phases 4 and 5. Although the presented use cases allow for illustrating the applicability of the derived methodological guidance across different healthcare IT phenotypes (i. e., self-management app, decision-support system) and target audiences (i. e., patients, case managers), they were selected purposefully and are therefore prone to subjectivity.

Future research to validate our methodological guidance

For this purpose, major evaluation studies comparing our approach with others head to head do not seem realistic. Instead, alternative means for validation are required. One possibility would be to have different projects carry out a user-centred design with different process models. These real-world projects could be scientifically accompanied in order to evaluate the extent of user participation and the usability of developed digital health solutions depending on the chosen process model.


With the introduced methodological guidance, we provide methodological guidance enabling technicians, health care professionals and other stakeholders interested in healthcare IT, to develop user-centred digital health interventions systematically. Applicants of our methodological guidance can make use of the guiding phases and the associated methods to assess and verify design decisions and iteratively evaluate their own intervention or product. The use cases presented highlight the utility of the blueprint as well as the methods assigned to its phases.

Scientifically monitored utilisation of our guidance in diverse settings may contribute to continuously improve its compatibility with the requirements of healthcare IT developers and the alignment with existing process models. Future research is needed to systematically enrich and prioritise the suggested methods and validate the proposed methodological guidance.


Appendix 1

Allocation of Phases, suggested methods and the corresponding research items.


0 State of the art research, including theoretical foundations for user acceptance [35, 44] Desk research

1 Analysing the user requirements [8, 32, 35, 37, 38, 86, 87, 88, 89, 90, 91] and the context of use [92, 93] Direct observations [32, 89]
Use cases [20, 35, 38]
Expert panels of patients and health care providers [90]
Workshops with patients [93]
Semi-structured and field interviews with patients [35, 37, 38, 88, 89, 90], their relatives, health care providers [32, 35, 37, 38] and hospital managers [37]
Quantitative surveys with patients [20, 91]
Focus groups with patients [32, 38, 87] and health care providers [87, 88, 93]
Personas and user profiles of patients [20, 32, 35, 86] and health care providers [35, 86]
Field studies [32, 88]

2 Concept development [38] and deduction of characteristics of the prototype [37, 86, 87, 88, 89, 91, 92] based on an in-depth understanding of the user requirements [8, 32, 35, 86, 88, 90, 92, 93] and context of use [8, 90] Use cases [38]
Focus groups with patients [38] and health care providers [37]
Semi-structured interviews with patients [38, 88] and health care providers [38]
Paper Prototyping or Sketching of the intended applications [20, 35, 88, 90]
Personas of patients [90]
Prototyping [32, 35, 86, 87, 88, 89]

3 Development [8, 32, 35, 38, 86, 87, 89, 90, 92, 93] and adaption [32, 86, 87] of the prototype based on a first evaluation [8, 20, 35, 37, 86, 87, 88, 89, 91, 92] of its usability in controlled environments Cognitive walkthrough with health care providers [20]
Heuristic evaluation with non-users [20, 93]
Think aloud studies with health care providers [20, 88, 92], patients [88, 92, 93] and their relatives [89]
Interview [87, 89, 90]
Quantitative surveys with patients [20, 37, 88, 90, 91, 93] and health care providers [20, 87, 88], using usability questionnaires

4 Evaluating [8, 32, 37, 38, 86, 89, 93] and adapting [37, 91, 93] the application in terms of usability and user acceptance under real-life conditions Field studies [88]
Focus groups with health care providers [37]
Quantitative surveys with patients [20, 37, 88, 90, 91, 93] and health care providers [20, 88], using usability questionnaires

5 Implementation [32, 35, 37, 91, 92] of the full-scale application and long-term evaluation [35, 37, 90, 93] Semi-structured interview with patients [38, 92] and health care providers [92]

Ethical and consent

Ethical statements were required for none of the use cases, however, we obtained clearance by our office for data security at the University Clinic Dresden for Use Case No. 2.

Funding information

The work on this paper was partly funded by the European Social Fund and the Free State of Saxony (Grant no. 100310385).

Competing Interests

The authors have no competing interests to declare.


  1. World Health Organisation (WHO). Innovative Care for Chronic Conditions: Building Blocks for Action [Internet]. 2002 [zitiert 29. Juli 2019]. Verfügbar unter: 

  2. Greenhalgh T, Wherton J, Sugarhood P, Hinder S, Procter R, Stones R. What matters to older people with assisted living needs? A phenomenological analysis of the use and non-use of telehealth and telecare. Social Science & Medicine. 1 September 2013; 93: 86–94. DOI: 

  3. Timpel P, Lang C, Wens J, Contel JC, Gilis-Januszewska A, Kemple K, u. a. Individualising Chronic Care Management by Analysing Patients’ Needs – A Mixed Method Approach. International Journal of Integrated Care. 13 November 2017; 17(6): 2. DOI: 

  4. Buse JB, Wexler DJ, Tsapas A, Rossing P, Mingrone G, Mathieu C, u. a. 2019 update to: Management of hyperglycaemia in type 2 diabetes, 2018. A consensus report by the American Diabetes Association (ADA) and the European Association for the Study of Diabetes (EASD). Diabetologia. 1 Februar 2020; 63(2): 221–8. DOI: 

  5. Sood S, Mbarika V, Jugoo S, Dookhy R, Doarn CR, Prakash N, u. a. What is telemedicine? A collection of 104 peer-reviewed perspectives and theoretical underpinnings. Telemedicine and e-Health. 2007; 13(5): 573–590. DOI: 

  6. Goyal S, Nunn CA, Rotondi M, Couperthwaite AB, Reiser S, Simone A, u. a. A Mobile App for the Self-Management of Type 1 Diabetes Among Adolescents: A Randomized Controlled Trial. JMIR Mhealth Uhealth. 19 Juni 2017; 5(6): e82. DOI: 

  7. Holmen H, Wahl AK, Cvancarova Smastuen M, Ribu L. Tailored Communication Within Mobile Apps for Diabetes Self-Management: A Systematic Review. Journal of medical Internet research. 23 Juni 2017; 19(6): e227. DOI: 

  8. Årsand E, Demiris G. User-centered methods for designing patient-centric self-help tools. Informatics for health & social care. September 2008; 33(3): 158–69. DOI: 

  9. Haas L, Maryniuk M, Beck J, Cox CE, Duker P, Edwards L, u. a. National Standards for Diabetes Self-Management Education and Support. Diabetes Care. 1 Januar 2014; 37(Supplement 1): S144–53. DOI: 

  10. American Diabetes Association. Standards of Medical Care in Diabetes—2016 Abridged for Primary Care Providers. Clin Diabetes. Januar 2016; 34(1): 3–21. DOI: 

  11. Lie SS, Karlsen B, Oord ER, Graue M, Oftedal B. Dropout From an eHealth Intervention for Adults With Type 2 Diabetes: A Qualitative Study. Journal of medical Internet research. Mai 2017; 19(5): e187–e187. DOI: 

  12. Mangin D, Parascandalo J, Khudoyarova O, Agarwal G, Bismah V, Orr S. Multimorbidity, eHealth and implications for equity: a cross-sectional survey of patient perspectives on eHealth. BMJ Open. 12 2019; 9(2): e023731. DOI: 

  13. Lyles CR, Allen JY, Poole D, Tieu L, Kanter MH, Garrido T. “I Want to Keep the Personal Relationship With My Doctor”: Understanding Barriers to Portal Use among African Americans and Latinos. Journal of medical Internet research. Oktober 2016; 18(10): e263–e263. DOI: 

  14. Garcia-Jimenez M, Calvo-Friedman A, Singer K, Tanner M. Patient Experiences with the Use of Telephone Interpreter Services: An Exploratory, Qualitative Study of Spanish-Speaking Patients at an Urban Community Health Center. Narrative inquiry in bioethics. 2019; 9(2): 149–62. DOI: 

  15. Otto L, Harst L. Bringing telemedicine into regular care: Theoretical underpinning for user-centred design processes. In: Proceedings of the Twenty-Third Pacific Asia Conference on Information Systems. Xi’an, China; 2019. 

  16. Harst L, Lantzsch H, Scheibe M. Theories Predicting End-User Acceptance of Telemedicine Use: Systematic Review. Journal of Medical Internet Research. 2019; 21(5): e13117. DOI: 

  17. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 29 September 2008; 337. DOI: 

  18. Norman DA, Draper SW. User Centered System Design; New Perspectives on Human-Computer Interaction. USA: L. Erlbaum Associates Inc.; 1986. DOI: 

  19. European Committee for Standardization. ISO 9241-210:2019(en), Ergonomics of human-system interaction — Part 210: Human-centred design for interactive systems [Internet]. 2011 [zitiert 15. November 2019]. Verfügbar unter: 

  20. Harte R, Glynn L, Rodríguez-Molinero A, Baker PM, Scharf T, Quinlan LR, u. a. A Human-Centered Design Methodology to Enhance the Usability, Human Factors, and User Experience of Connected Health Systems: A Three-Phase Methodology. JMIR Human Factors. 2017; 4(1): e8. DOI: 

  21. González-Ortiz LG, Calciolari S, Goodwin N, Stein V. The Core Dimensions of Integrated Care: A Literature Review to Support the Development of a Comprehensive Framework for Implementing Integrated Care. Int J Integr Care. 2018; 18(3). DOI: 

  22. Goodwin N. Improving Integrated Care: Can Implementation Science Unlock the ‘Black Box’ of Complexities? Int J Integr Care. 2019; 19(3). DOI: 

  23. Sanders EBN. From user-centered to participatory design approaches. In: Design and the Social Sciences: Making Connections. CRC Press; 2002. 

  24. Swinkels ICS, Huygens MWJ, Schoenmakers TM, Oude Nijeweme-D’Hollosy W, van Velsen L, Vermeulen J, u. a. Lessons Learned From a Living Lab on the Broad Adoption of eHealth in Primary Health Care. J Med Internet Res. 29 März. DOI: 

  25. Esser PE, Goossens RH. A framework for the design of user-centred teleconsulting systems. Journal of telemedicine and telecare. 2009; 15(1): 32–9. DOI: 

  26. Taylor HA, Sullivan D, Mullen C, Johnson CM. Implementation of a user-centered framework in the development of a web-based health information database and call center. Journal of Biomedical Informatics. 1 Oktober 2011; 44(5): 897–908. DOI: 

  27. Schnall R, Rojas M, Bakken S, Brown W, Carballo-Dieguez A, Carry M, u. a. A user-centered model for designing consumer mobile health (mHealth) applications (apps). Journal of Biomedical Informatics. 1 April 2016; 60: 243–51. DOI: 

  28. Wodchis WP, Ashton T, Baker GR, Sheridan N, Kuluski K, McKillop A, u. a. A Research Program on Implementing Integrated Care for Older Adults with Complex Health Needs (iCOACH): An International Collaboration. Int J Integr Care. 2018; 18(2). DOI: 

  29. Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Info Libr J. Juni 2009; 26(2): 91–108. DOI: 

  30. Mayring P. Qualitative Content Analysis. Forum: Qualitative Social Research. 2000; 2(1): 1–10. 

  31. Sutcliffe A, Thew S, De Bruijn O, Buchan I, Jarvis P, McNaught J, u. a. User engagement by user-centred design in e-Health. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences. 13 September 2010; 368(1926): 4209–24. DOI: 

  32. LeRouge C, Ma J, Sneha S, Tolle K. User profiles and personas in the design and development of consumer health technologies. International Journal of Medical Informatics. 1 November 2013; 82(11): e251–68. DOI: 

  33. Hennemann S, Beutel ME, Zwerenz R. Drivers and Barriers to Acceptance of Web-Based Aftercare of Patients in Inpatient Routine Care: A Cross-Sectional Survey. Journal of medical Internet research. 23 Dezember 2016; 18(12): e337. DOI: 

  34. Wöckl B, Yildizoglu U, Buber I, Aparicio Diaz B, Kruijff E, Tscheligi M. Basic senior personas: a representative design tool covering the spectrum of European older adults. In: Proceedings of the 14th international ACM SIGACCESS conference on Computers and accessibility – ASSETS ’12. Boulder, Colorado, USA: ACM Press; 2012. S. 25. DOI: 

  35. Gemert-Pijnen JE van, Nijland N, Limburg M van, Ossebaard HC, Kelders SM, Eysenbach G, u. a. A Holistic Framework to Improve the Uptake and Impact of eHealth Technologies. Journal of Medical Internet Research. 2011; 13(4): e111. DOI: 

  36. Elavsky S, Smahel D, Machackova H. Who are mobile app users from healthy lifestyle websites? Analysis of patterns of app use and user characteristics. Translational behavioral medicine. Dezember 2017; 7(4): 891–901. DOI: 

  37. Gray CS, Gravesande J, Hans PK, Nie JX, Sharpe S, Loganathan M, u. a. Using Exploratory Trials to Identify Relevant Contexts and Mechanisms in Complex Electronic Health Interventions: Evaluating the Electronic Patient-Reported Outcome Tool. JMIR Formative Research. 2019; 3(1): e11950. DOI: 

  38. van der Weegen S, Verwey R, Spreeuwenberg M, Tange H, van der Weijden T, de Witte L. The Development of a Mobile Monitoring and Feedback Tool to Stimulate Physical Activity of People With a Chronic Disease in Primary Care: A User-Centered Design. JMIR mHealth and uHealth. 2013; 1(2): e8. DOI: 

  39. Snyder C, Safari an OMC. Paper Prototyping. Burlington, Massachusetts: Morgan Kaufmann; 2003. 

  40. Bolle S, Romijn G, Smets EMA, Loos EF, Kunneman M, van Weert JCM. Older Cancer Patients’ User Experiences With Web-Based Health Information Tools: A Think-Aloud Study. J Med Internet Res. 25 2016; 18(7): e208. DOI: 

  41. Georgsson M, Staggers N, Årsand E, Kushniruk A. Employing a user-centered cognitive walkthrough to evaluate a mHealth diabetes self-management application: A case study and beginning method validation. J Biomed Inform. März 2019; 91: 103110. DOI: 

  42. Lewis JR. IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. International Journal of Human–Computer Interaction. 1 Januar 1995; 7(1): 57–78. DOI: 

  43. Agnisarman S, Narasimha S, Madathil KC, Welch B, Brinda FNU, Ashok A, u. a. Toward a More Usable Home-Based Video Telemedicine System: A Heuristic Evaluation of the Clinician User Interfaces of Home-Based Video Telemedicine Systems. JMIR Human Factors. 2017; 4(2): e11. DOI: 

  44. Kabeza CB, Harst L, Schwarz PEH, Timpel P. Assessment of Rwandan diabetic patients’ needs and expectations to develop their first diabetes self-management smartphone application (Kir’App). Therapeutic Advances in Endocrinology. 1 Januar 2019; 10: 2042018819845318. DOI: 

  45. Timpel P, Oswald S, Schwarz PEH, Harst L. Mapping the Evidence on the Effectiveness of Telemedicine Interventions in Diabetes, Dyslipidemia, and Hypertension: An Umbrella Review of Systematic Reviews and Meta-Analyses. Journal of Medical Internet Research. 2020; 22(3): e16791. DOI: 

  46. Kabeza CB, Harst L, Schwarz PEH, Timpel P. A qualitative study of users’ experiences after 3 months – the first Rwandan diabetes self-management Smartphone application “Kir’App”. Therapeutic Advances in Endocrinology and Metabolism; 2019. DOI: 

  47. Evans R, Brockman R, Grey J, Bell S, Harding S, Gunnell D, u. a. A cluster randomised controlled trial of the Wellbeing in Secondary Education (WISE) Project – an intervention to improve the mental health support and training available to secondary school teachers: protocol for an integrated process evaluation. Trials. 4 Mai 2018. DOI: 

  48. Schoultz M, Macaden L, Hubbard G. Participants’ perspectives on mindfulness-based cognitive therapy for inflammatory bowel disease: a qualitative study nested within a pilot randomised controlled trial. Pilot Feasibility Stud. 19 Januar 2016. DOI: 

  49. Hayashi A, Yamaguchi S, Waki K, Fujiu K, Hanafusa N, Nishi T, u. a. Testing the Feasibility and Usability of a Novel Smartphone-Based Self-Management Support System for Dialysis Patients: A Pilot Study. JMIR research protocols. 20 April 2017; 6(4): e63. DOI: 

  50. OECD. Health at a Glance 2019 [Internet]. 2019. Verfügbar unter: 

  51. Miani C, Ball S, Pitchforth E, Exley J, King S, Roland M, u. a. Organisational interventions to reduce length of stay in hospital: a rapid evidence assessment; 2014. DOI: 

  52. Bankston-White C, Birmingham J. Case Management Directors: How to Manage in a Transition-Focused World: Part 1. Professional Case Management. April 2015; 20(2): 63–78. DOI: 

  53. Åhsberg E. Discharge from hospital – a national survey of transition to out-patient care. Scand J Caring Sci. Juni 2019; 33(2): 329–35. DOI: 

  54. Mehta RL, Baxendale B, Roth K, Caswell V, Le Jeune I, Hawkins J, u. a. Assessing the impact of the introduction of an electronic hospital discharge system on the completeness and timeliness of discharge communication: a before and after study. BMC Health Serv Res. 5 September 2017; 17. DOI: 

  55. Brooke J. SUS: A “Quick and Dirty” Usability Scale. In: Usability Evaluation In Industry. 1. Aufl. London: CRC Press; 1996. 

  56. Maeder AJ, Williams PAH. Health Smart Homes: New Challenges. Studies in Health Technology and Informatics. 2017; 166–169. DOI: 

  57. Haux R, Koch S, Lovell NH, Marschollek M, Nakashima N, Wolf K-H. Health-Enabling and Ambient Assistive Technologies: Past, Present, Future. Yearbook of Medical Informatics. August 2016; 25(S 01): S76–91. DOI: 

  58. Bekiaris A, Mourouzis A, Maglaveras N. The REMOTE AAL Project: Remote Health and Social Care for Independent Living of Isolated Elderly with Chronic Conditions. In: Stephanidis C, Herausgeber (ed.). Universal Access in Human-Computer Interaction Context Diversity. Berlin, Heidelberg: Springer; 2011. S. 131–40. (Lecture Notes in Computer Science). DOI: 

  59. Wollschlaeger B, Kabitzsch K. Navigating the Jungle of Assistance Systems: A Comparison of Standards for Assistance Functionality. In: Proceedings of the 12th International Joint Conference on Biomedical Engineering Systems and Technologies – Volume 5: HEALTHINF. SciTePress; >2019. S. 359–366. DOI: 

  60. Wollschlaeger B, Kabitzsch K. Automated Engineering for Health Smart Homes: Find a Way in the Jungle of Assistance Systems. Stud Health Technol Inform. 16 Juni 2020; 270: 828–32. DOI: 

  61. Debong F, Mayer H, Kober J. Real-World Assessments of mySugr Mobile Health App. Diabetes Technol Ther. 2019; 21(S2): S235–40. DOI: 

  62. Perimal-Lewis L, Maeder A, Gordon S, Tieman J. A Tablet-Based Memory Enhancement Application for Older Users: Design Approach. Hawaii International Conference on System Sciences; 2020. DOI: 

  63. Nadal C, Doherty G, Sas C. Technology acceptability, acceptance and adoption – definitions and measurement. In: Conference on Human Factors in Computing Systems (CHI’19) Symposium: Workgroup on Interactive Systems in Healthcare (WISH). ACM; 2019. 

  64. User-Centered Design Basics [Internet]. Department of Health and Human Services; 2017 [zitiert 25. März 2020]. Verfügbar unter: /what-and-why/user-centered-design.html. 

  65. Greenbaum J, Kyng M. Design at work cooperative design of computer systems. Hillsdale, NJ [u.a.]: Lawrence Erlbaum; 1991. 

  66. Wagner EH, Austin BT, Von Korff M. Organizing Care for Patients with Chronic Illness. The Milbank Quarterly. 1996; 74(4): 511–44. DOI: 

  67. Steele Gray C, Wodchis WP, Baker GR, Carswell P, Kenealy T, McKillop A, u. a. Mapping for Conceptual Clarity: Exploring Implementation of Integrated Community-Based Primary Health Care from a Whole Systems Perspective. Int J Integr Care. 21 März 2018; 18(1): 14. DOI: 

  68. Roberts A, Philip L, Currie M, Mort A. Striking a balance between in-person care and the use of eHealth to support the older rural population with chronic pain. Int J Qual Stud Health Well-being. 2015; 10: 27536. DOI: 

  69. Giguere AMC, Lawani MA, Fortier-Brochu E, Carmichael PH, Legare F, Kroger E, u. a. Tailoring and evaluating an intervention to improve shared decision-making among seniors with dementia, their caregivers, and healthcare providers: study protocol for a randomized controlled trial. Trials. 25 Juni 2018; 19(1): 332. DOI: 

  70. Dalton H, Read DMY, Booth A, Perkins D, Goodwin N, Hendry A, u. a. Formative Evaluation of the Central Coast Integrated Care Program (CCICP), NSW Australia. Int J Integr Care. 2019; 19(3). DOI: 

  71. van der Heide I, Snoeijs S, Quattrini S, Struckmann V, Hujala A, Schellevis F, u. a. Patient-centeredness of integrated care programs for people with multimorbidity. Results from the European ICARE4EU project. Health Policy. 2018; 122(1): 36–43. DOI: 

  72. Timpel P, Harst L. Research Implications for future telemedicine studies and innovations. Eur J Public Health. 1 November 2019; 29(Supplement_4). DOI: 

  73. Langbecker D, Caffery LJ, Gillespie N, Smith AC. Using survey methods in telehealth research: A practical guide. J Telemed Telecare. Oktober 2017; 23(9): 770–9. DOI: 

  74. Greenhalgh T, Wherton J, Papoutsi C, Lynch J, Hughes G, A’Court C, u. a. Beyond Adoption: A New Framework for Theorizing and Evaluating Nonadoption, Abandonment, and Challenges to the Scale-Up, Spread, and Sustainability of Health and Care Technologies. Journal of Medical Internet Research. 2017; 19(11): e367. DOI: 

  75. Morozov S, Guseva E, Ledikhova N, Vladzymyrskyy A, Safronov D. Telemedicine-based system for quality management and peer review in radiology. Insights Imaging. 1 Juni 2018; 9(3): 337–41. DOI: 

  76. Read DMY, Dalton H, Booth A, Goodwin N, Hendry A, Perkins D. Using the Project INTEGRATE Framework in Practice in Central Coast, Australia. Int J Integr Care. 2019; 19(2). DOI: 

  77. van Dyk L, Schutte CSL. Development of a maturity model for telemedicine. South African Journal of Industrial Engineering. Juli 2012; 23(2): 61–72. DOI: 

  78. Rogers EM. Diffusion of innovations. 5th edition. New York: Free Press; 2003. XXI, 551 S.: Ill., graph. Darst., Kt. 

  79. Hastall M, Dockweiler C, Mühlhaus J. Achieving end user acceptance: Building blocks for an evidence-based user-centered framework for health technology development and assessment. Universal Access in Human–Computer Interaction Human and Technological Environments [Internet]. 2017 [zitiert 3. Januar 2018]; Verfügbar unter: DOI: 

  80. Boehm BW. A spiral model of software development and enhancement. Computer. Mai 1988; 21(5): 61–72. DOI: 

  81. Schwaber K. SCRUM Development Process. In: Sutherland J, Casanave C, Miller J, Patel P, Hollowell G, Herausgeber (eds.), Business Object Design and Implementation. London: Springer; 1997. S. 117–34. DOI: 

  82. Ruparelia NB. Software development lifecycle models. ACM SIGSOFT Software Engineering Notes. 11 Mai 2010. DOI: 

  83. van den Kieboom RC, Bongers IM, Mark RE, Snaphaan LJ. User-Driven Living Lab for Assistive Technology to Support People With Dementia Living at Home: Protocol for Developing Co-Creation–Based Innovations. JMIR Res Protoc. 28 Januar 2019. DOI: 

  84. National Institute for Health and Care Excellence. Evidence standards framework for digital health technologies. NICE; 2019. 

  85. Southam-Gerow MA, Dorsey S. Qualitative and Mixed Methods Research in Dissemination and Implementation Science: Introduction to the Special Issue. Journal of Clinical Child & Adolescent Psychology. 2 November 2014; 43(6): 845–50. DOI: 

  86. Martínez-Alcalá CI, Muñoz M, Monguet-Fierro J. Design and Customization of Telemedicine Systems. Computational and Mathematical Methods in Medicine; 2013. DOI: 

  87. Maurik IS van, Visser LN, Pel-Littel RE, Buchem MM van, Zwan MD, Kunneman M, u. a. Development and Usability of ADappt: Web-Based Tool to Support Clinicians, Patients, and Caregivers in the Diagnosis of Mild Cognitive Impairment and Alzheimer Disease. JMIR Formative Research. 2019; 3(3): e13417. DOI: 

  88. McCurdie T, Taneva S, Casselman M, Yeung M, McDaniel C, Ho W, u. a. mHealth Consumer Apps: The Case for User-Centered Design. Biomedical Instrumentation & Technology. 1 Januar 2012; 46(s2): 49–56. DOI: 

  89. Taylor DP, Bray BE, Staggers N, Olson RJ. User-Centered Development of a Web-Based Preschool Vision Screening Tool. AMIA Annu Symp Proc. 2003; 2003: 654–8. 

  90. Vilardaga R, Rizo J, Zeng E, Kientz JA, Ries R, Otis C, u. a. User-Centered Design of Learn to Quit, a Smoking Cessation Smartphone App for People With Serious Mental Illness. JMIR Serious Games. 2018; 6(1): e2. DOI: 

  91. Zhou L, DeAlmeida D, Parmanto B. Applying a User-Centered Approach to Building a Mobile Personal Health Record App: Development and Usability Study. JMIR mHealth and uHealth. 2019; 7(7): e13194. DOI: 

  92. Catwell L, Sheikh A. Evaluating eHealth Interventions: The Need for Continuous Systemic Evaluation. PLOS Medicine. 18 August 2009; 6(8): e1000126. DOI: 

  93. Vermeulen J, Neyens JC, Spreeuwenberg MD, van Rossum E, Sipers W, Habets H, u. a. User-centered development and testing of a monitoring system that provides feedback regarding physical functioning to elderly people. Patient Prefer Adherence. 27 August 2013; 7: 843–54. DOI: