Joint inspection of services for people with learning disabilities in Scotland: compliance or commitment?
International Journal of Integrated Care, 5 October 2006 - ISSN 1568-4156
Projects and Developments
Joint inspection of services for people with learning disabilities in Scotland: compliance or commitment?
Martin Campbell, School of Psychology, University of St Andrews
Correspondence to: Martin Campbell, University of St Andrews, St Andrews KY16 9JP, Phone: +1334 462084, Fax: +1334 463042, E-mail: mc1@st-andrews.ac.uk
Abstract


Purpose: The article describes the development of a practical model of joint, integrated inspection of managed care services for people with learning disabilities in Scotland. The model will give a reliable measure of the impact services are making to people's lives and the quality of service that individuals are actually receiving.


Context of case: At present health, social services and education services for people with learning disabilities in Scotland are inspected separately, by up to nine different agencies. The first joint, integrated inspections of all services for people with learning disabilities in Scotland will take place in 2006. This is the first inspection of its kind in the UK, and the first to involve carers and people with learning disabilities on the inspection team.


Data sources: Quality Outcome Indicators were developed in 21 different areas, or domains. Evidence based best practice, and evaluative data from previous inspections were the primary sources of data.


Case description: This paper reviews the background and rationale for the integrated, joint inspection process. Strengths and constraints of this approach to inspection are discussed, including the crucial importance of commitment from services and from inspectors, rather than mere compliance with demands. Some guidance on how to fully involve staff, carers and services users in the inspection process is given.


Conclusions and discussion: The model will produce data to inform decision-making for managers in integrated services and give services users clear information about how well local needs are being met, what areas need development, and what capacity the organisations have to improve. The model of inspection may be of interest to practitioners in a national and international context. The model will be evaluated, following the first joint inspection.

Keywords
quality of care; clinical efficiency; client perspective; integrated care
Introduction
Background

There are currently 32 Local Authority areas in Scotland, providing Social Work and Education services, and 15 Health Boards, providing primary and community health services. Health services in the UK are administered by a range of health authorities and health boards. There are 100 health authorities in England and five in Wales, 15 health boards in Scotland and four joint health and social services boards in Northern Ireland. There are some differences between the NHS in Scotland, in the context of the Scottish government [1], and the NHS in England [2], although both systems are funded essentially in the same way [3].

The NHS in Scotland (NHSiS) was formed by separate legislation, the NHS (Scotland) Act 1947 and has a separate identity within the UK. Ministries responsible for Health and the NHS were set up after devolution and the setting up of the Scottish Parliament in 1999. Devolving management and health spending in 1999 to the newly created Scottish Parliament further reinforced Scotland's separate identity in a UK context.

There were historical differences in the development of services in England, Scotland, Wales and Northern Ireland, before the setting up of the Scottish Parliament in 1999 [4].

Official regulation and inspection of health and social services in the UK is more evident than in some other European countries. In England health services are inspected by the newly formed Healthcare Commission [5]; the social care services by the Commission for Social Care Inspection [6]. There is a common framework for the development of quality and clinical governance for the NHS in Scotland and England, in the context of integrated care [7].

Managed care services for people with learning disabilities are currently inspected by up to nine different regulatory organisations in Scotland. In December 2003 Peter Peacock, Minister of Scottish Parliament, Scottish Minister for Education and Young People said that there should be joint, integrated inspection of services for all people with learning disabilities in Scotland. In May 2004 a Joint Inspection Steering Group was set up to look at the best ways of making joint, integrated inspections work. The Joint Inspection Steering Group was chaired by the Social Work Inspection Agency (SWIA) [8] and had membership from inspection organisations including: The Commission for the Regulation of Care, National Health Service Quality Improvement Scotland (NHS QIS), Mental Welfare Commission (MWC), Audit Scotland, Communities Scotland, Her Majesty's Inspectorate for Education (HMIE), Her Majesty's Inspectorate of Prisons (HMIP), Her Majesty's Inspectorate of Constabulary (HMIC), and Communities Scotland. Also represented were carers and people with learning disabilities from People First Scotland and Carers Scotland. In 2005 there was a multi-disciplinary national conference about Joint Inspection, “Inspecting Together Making a Difference” and the Joint Inspection Steering Group, commissioned a model of Joint Inspection from the University of St Andrews. Between February–April 2006 a draft model of joint inspection was put out for consultation. The first joint inspection of services for people with learning disabilities in Scotland took place in June 2006.

Getting it right

There should be good services for people with learning disabilities. To ensure that this is the case, services for people with learning disabilities in Scotland are currently inspected in a number of different ways, by a number of different government organisations; The Social Work Inspection Agency (SWIA) inspect social services, Her Majesty's Inspectorate of Education (HMIe) inspect schools and educational organisations; National Health Service Quality Improvement Scotland (NHSQIS) inspect in-patient and community health services. In addition, the Scottish Commission for the Regulation of Care (Care Commission) inspect and register a variety of social care, health and voluntary organisations. Because of the many agencies and methods involved, some services are over inspected, being asked to produce the same information in different formats, placing an unnecessary administrative burden on staff and additional resource implications in spending time with regulatory/inspection staff. Some services are not inspected regularly, and a few services are not inspected at all.

The original rationale for inspecting services for people with learning disabilities in Scotland, and in the UK generally, is found in a history of government polices, which try to both protect against neglect and abuse, and to ensure effective performance management. For example, organised inspection of health services dates back to the Scottish Hospital Advisory Service, established in 1970, while inspection of schools by Her Majesty's Inspectorate of Education has now been happening for more than one hundred years. Joint working between health and social care partners is slowly developing in Scotland and integrated inspections have come about in this context. Scotland has health, social work and education services that are different from, and independent of those in the rest of the UK.

It is generally agreed that inspection should focus on outcomes of policy and practice, and show clearly how people's lives are improved. A new model of integrated, joint inspection has been developed, including a set of outcome indicators that are applicable to all people with learning disabilities in Scotland, to cover multiple service settings, e.g. hospitals, community services, day services or family homes.

Inspecting services should improve them. Inspectors, staff in services and people with learning disabilities and their carers should agree on the best ways to do inspection, and should all be involved in the actual inspections. The overall aim of inspection is to help people with learning disabilities to have a better quality of life (e.g. [912]). Quality of life for people with or without learning disabilities is usually measured in terms of independence, self-determination, choice-making and exercising personal control [10,1315].

A key, unresolved question for joint inspection, however, is whether the inspection process is driven by compliance or by commitment. This applies both to those being inspected as to the inspectors. Are services complying with imposed periodic standards and outcome indicators, or are they committed to the same periodic inspection, linked to on-going monitoring and self evaluation? Similarly, are the independent social work, health and education inspectorates complying with government policy, or have they fully committed to joint inspection, with the possible loss of some of their individual inspectorate identity and authority?

A paper by Hatton [16] on developing outcome-based Performance Indicators explores how the joint inspection process may evolve in England and Wales, and explores some of the issues discussed in this paper. The author is grateful to Chris Hatton and his colleagues for support and advice in developing the Scottish model of joint inspection.

Method

In December 2005 the author was commissioned by the Scottish Joint Inspection Steering Committee to produce a model of joint inspection that combined elements from three existing approaches to inspection and review, namely:

  1. a Performance Inspection model used by Social Work Inspection Agency

  2. a National Care Standards model used by the Care Commission

  3. a Quality Indicators model used by NHS Quality Improvement Scotland

Aspirations

There are a number of important assumptions made in devising a model of joint inspection. Accepting that the process will not be perfect, the following aspirational goals were included:

  • People with learning disabilities should benefit practically and significantly from health, social and education services

  • Services have a legal and moral obligation to provide a consistently high standard of care and support and to achieve outcomes

  • The joint inspection process should be clear, adequate and feasible

  • The joint inspection model will be designed specifically for services to people with learning disability; other care groups will need a different model

  • Outcome indicators are a valid method of evaluating the performance of services

  • The list of quality outcome indicators used to inspect services will not be comprehensive. However, the methodology used and the range of indicators used will be sufficient to give a reliable overall measure of the quality across the services

  • Evaluation of services is necessary to ensure that evidence of best practice and outcomes is generated regularly and helps services to continuously improve services for all service users

  • Staff in services have the skills and training to achieve agreed outcomes for service users in their service

  • Plain English words should be used so that all stakeholders can easily understand the model of joint inspection

  • General indicators about satisfaction with services and life in general are not outcome indicators, and research has demonstrated that measures of satisfaction (or subjective well-being) are relatively insensitive to changes in people's circumstances, living conditions or lifestyle [17]

  • The joint inspection of services for people with learning disabilities will exclude those services already registered and inspected under The Regulation of Care (Scotland) Act 2001

The challenge of developing a model for joint inspection is to make that model complex enough to meet the need, but simple enough that it can be understood. “Joint Inspection Bingo” (Figure 1) is an attempt to summarise and simplify the main requirements of joint inspection. Five in a row, either across, down or diagonally is a good start but a “full house” is needed if all of the requirements are to be met! This Bingo card, with general headings, was used as a useful starting point for discussing elements of the model with service managers, carers and people with learning disabilities.

From these general headings a set of suitable outcome indicators was developed, to give agreed measures of quality.

Why use outcome indicators?

An Outcome Measure is the desired or ideal impact of a service, i.e. what change it hopes to make to a person or a group of people with learning disabilities. An Outcome Indicator is the actual impact made by the activities of a service or services. Outcome indicators assess the extent to which part of a service or the service as a whole achieves its stated objectives or outcome measures for people with learning disabilities [1821].

The outcome indicators to be used should, ideally, have all of the following features. They should:

  • be relevant to current government policies

  • have been used previously with people with learning disabilities or family members

  • allow for comparisons to Scottish general population data (adapted from [22])

These three criteria were used to guide the choice of the final outcome indicators selected for the model.

Research consistently shows that service outputs and processes (e.g. costs, resources used, staff ratios) generally have weak or no associations with actual outcomes for service users [23]. Outcomes identified by service users and carers should be central to any model of inspection. Outcome indicators also provide more clarity for staff by specifically linking the work they do with outcomes it is designed to achieve [24].

Quality of life is increasingly used as a basis for policies and practices in the field of learning disabilities [25]. Quality of life can vary between areas and according to what each person values, but there are a number of established outcomes derived from research and application over the past twenty years [10]. Quality of life for most people with learning disabilities is influenced by a combination of the quality of services and quality of care they receive (Figure 2). For people with more complex disabilities quality of life may be almost wholly determined by a combination of care and services received. Some services have moved from measuring simply quality of care to the wider concept of quality of life [26].

Quality of care is a measure of staff performance. For example, how well people are doing their job. Quality of service is a measure of service effectiveness [27,28]. For example, how many of the service users actually receive an effective service? Quality of life can be used as a framework for assessing outcomes. Quality of life outcomes, in the context of joint inspection, give information about how a person's life has changed directly as a result of receiving a particular service.

In practice, the three may be related (see Figure 2).

If the outcomes indicators are used across a range of services and provide some kind of “benchmarking”, [28] then it is crucial that the measures used are value based, in line with the national policy. Outcome indicators should ideally also have good psychometric properties—i.e. there should be some evidence of their reliability and validity, and they should be sensitive to detecting changes over time that are relevant to users and carers (e.g. [10,2931]). Previously used outcome indicators have been selected to improve reliability and validity, but this can only be properly assessed by evaluation, post inspection.

What information is needed and how will it be collected?

There is a balance needed between what information it would be interesting to collect and what it is possible and feasible for services to systematically collect, without the collection process itself adversely affecting the quality of services. If the data collected are to reliably reflect the quality of life of service users then it will be a combination of case record audit, review of policy and procedure documents and responses from services users about the quality of services. Information collected directly from service users will be vital:

A substantial proportion of relevant outcome data will need to be collected (or is certainly most efficiently collected) from service users themselves or, for some, from proxy respondents, and cannot be derived from service activity data [16].

The outcome indicators selected have been grouped into 21 different areas or domains (see Figure 3). These have been developed from a number of sources, including Social Work Inspection Agency, Care Commission, NHS Quality Improvement Scotland, The Quality Network (British Institute of Learning Disabilities/National Development Team), National Core Indicators Project, National Survey of Adults with Learning Difficulties in England (UK), Child Family Survey—postal questionnaire with families with a child with developmental disabilities, ASF Adult Family Survey—postal questionnaire with families with an adult with developmental disabilities living with them, The Quality Network (British Institute of Learning Disabilities/National Development Team). Joint inspection outcome indicators have been based on questions that services would like answered and outcome indicators that service users would like answered.

The first group of outcome indicators (1–11) are assessed using the triangulation methodology (Figure 4). The second group of indicators (A–J) are based on a specific set of questions put to a representative sample of service users, family members and supporters, in interview before and during joint inspection. A full set of the Outcome Indicators is available on request from the author. This information will be used to verify information collected elsewhere, and to identify other areas of service provision not covered by the questions in 1–11. Some questions can only be answered by service users and their carers; some questions can only be answered by staff, at various levels, in services (Figure 3).

In summary, it should be possible to collect the data for outcome indicators feasibly, without an increase in the demands made on users or staff by the current inspection process (e.g. [32,33]). Some of the information to be analysed by the joint inspection team will already be available from service providers or from other inspection organisations; some of the information will be new and will need to be collated by services.

Joint inspection—practical aspects

Joint inspection teams are multidisciplinary, with representatives in all major inspectorates. There are also people who have learning disabilities and carers on each team. All teams will be led by an experienced professional from one of the inspection organisations. One inspection agency will take a lead role in the joint inspection. This need not be the same agency for each inspection. The joint inspection team will base a report on information gathering, fieldwork and analysis.

In summary the basic process of joint inspection will be as follows:

  • Outcome indicators have been agreed, and each Local Authority/National Health Service area to be inspected is asked to undertake a self-evaluation of its service using the outcomes indicators as a template. In the planning phase, Joint Inspectors use inspection information that is already held by other regulators and inspectorates, and will not seek information from service providers that is already published or publicly available

  • Inspectors review all documents supplied, audit social work, education and health records of people with learning disabilities, either directly or by proxy and look for patterns in information supplied by people with learning disabilities, family carers and staff

  • The joint inspection team may look in more depth at particular service areas, chosen in the light of analysis of information gained

  • The joint inspection report will assess the closeness of the match between the self-evaluation of outcomes and the information collected during external inspection

The joint inspection process should be clear to all involved, it should be adequate to do the job and it should be practically feasible.

During the on-site inspection information from as many relevant sources as possible will be collected. Figure 4 shows the “triangulation” methodology. Joint inspection will triangulate information about a sample of service users. This will be done by interviewing the person, scrutinising his/her case notes, and checking how policies and procedures have impacted on the quality of his/her life. This methodology will give a more holistic view of how well that person's needs are being met by services.

The Social Work Inspection Agency (SWIA) currently uses six categories to describe performance. These categories, with some modifications, have been adopted for the joint inspection process to rate each of the outcome indicators. The suggested categories and the criteria for each are summarised in Figure 5. Extended criteria are also available to services being inspected and inspectors. “Smiley faces” are added to make the categories easier to understand.

Each category is judged in response to the question, “To what extent was the outcome achieved?” (Figure 5).

Discussion

From previous, single agency inspections and the pilot phase of joint inspection, it is apparent that the commitment, rather than the compliance of service users and carers in this process will be improved if clear information is supplied by inspectors about the purpose of collecting the data and the uses to which any results will be put.

Commitment to inspection will involve a fundamental culture shift in many services, where the link between self-evaluation of services and inspection is not commonly made.

The outcome indicators used form the basis of the final joint inspection report. The report will be presented in a form that should help decision-making for service managers and give services users clear information about how well local needs are being met, what areas need development, and what capacity the organisation has to improve. The report is a combination of questions that services would like answered (e.g. Are particular services effective? What kinds of services are needed?) and questions that service users would like answered (e.g. How are service users involved in planning? How well are services supporting people? Is information in easy read language and other formats?) [32,34,35].

The inclusion of people who have learning disabilities and carers in the development of the model and on each inspection team has made a significant difference to the joint inspections, and in one way has made the inspection truly “integrated”. During pilot joint inspections in 2003, four initial misgivings that their inclusion would be “tokenistic” in such a complex process, were unfounded. They have added a perspective and a credibility to the joint inspections that no professional can. The full impact of this is currently being evaluated separately.

Constraints to success of joint inspection

Joint inspection of learning disability services in Scotland is new and complex. Significant amounts of time and resources from all the inspectorates involved will be needed to ensure success. The first series of inspections will be used to establish a baseline for subsequent joint inspections. Some of the potential difficulties are summarised below. The constraints are listed according to how serious a threat they pose to the success of the joint inspection process, beginning with the most serious threats.

  1. To establish impact and changes directly attributable to services, it is necessary to begin with base-line data so that any change can be identified. For a new, joint inspection model and new outcome measures, there may not be any baseline data for comparison.

  2. The joint inspection will bring together inspection methodology and information from three different inspectorates, who carry out both area and unit level inspections. For the purposes of joint inspection and sharing intelligence it will be necessary to aggregate information from specified areas.

  3. Joint inspection must be carried out without compromising the existing legal duties or the independence of each of the separate organisations involved, e.g. The Regulation of Care (Scotland) Act 2001. Within each organisation the commitment to the joint inspection process is tempered by some anxiety about being absorbed into some future “Joint Inspectorate” body. It may be necessary to clarify the continuing role of each of the individual organisations in the context of joint inspection.

  4. Interviewing people with learning disabilities, carers and staff requires a range of complex skills. To ensure that information is collected in a consistent and reliable way, training will be needed for all inspectors. Issues here involve assessing the capacity of the person to answer questions and ensuring questions are accessible to the maximum number of people with learning disabilities.

  5. For people with profound and multiple disabilities, information about their quality of life will primarily be collected through interviews with carers. Again, consistency will be important. Clear criteria will be needed for deciding when to interview the person with learning disabilities and when to interview carers or supporters.

  6. Many staff still see inspection and quality assurance as activities that are imposed from above at certain times of the year, but not embedded in or related to delivery, and not contributing practically to service outcomes. Staff, understandably, can be suspicious when others who are not directly involved with service users provide evidence of service effectiveness, or lack of it.

  7. Additional outcomes measures may need to be developed for specific service users, e.g. Autistic Spectrum Disorder, people with mental health problems, particular ethnic groups.

Several of these constraints will be overcome by a genuine commitment to make the process work, on the part of inspectors and services; others will require a detailed evaluation of the joint inspection process itself, to ensure that it is more effective than independent inspections by each of the agencies, as at present.

In summary, collecting outcome information for an annual, one-off integrated inspection can be driven by compliance or by commitment. Making the systematic collection of outcomes measures part of everyday practice however, can only ever be maintained by commitment of the service staff at all levels. Rates of form filling and other data collection, as well as enthusiasm, will tend to fall off dramatically following service inspection and reviews (e.g. [36]).

When the joint inspection process is well established it is envisaged that the role of inspection will be principally one of verification, rather than inspection, of the self-evaluation information and the methods being used by services.

The model of integrated joint inspection described in this paper will be evaluated following the first inspection. It is hoped that the strategic and operational lessons learned from this evaluation will be of interest both to Scottish services and to services elsewhere, who may be developing integrated inspection models to ensure quality in their own managed care services.

Reviewers

Marian Maaskant, PhD, research co-ordinator with ‘Pepijn en Paulus’ at Echt (NL), researcher Department of Health Care Studies/Governor Kremers Centre of the University of Maastricht, Netherlands.

Roy McConkey, Professor, School of Nursing, University of Ulster, Newtownabbey, N. Ireland

Alison Petch, Director, research in practice for adults, Dartington Hall Trust, Devon, UK

References
1.
Audit Scotland. How government works in Scotland: A report to the Scottish Parliament by the Auditor General for Scotland [Online]. Edinburgh: 2002. Available from: URL: http://www.audit-scotland.gov.uk/publications/pdf/2002/02ar04ag.pdf.
2.
Department of Health. About the NHS—How the NHS Works in England [Online].. Available from: URL: http://www.nhs.uk/england/AboutTheNhs/Default.cmsx.
3.
British Medical Association. Briefing on the funding of the NHS in Scotland: Healthcare funding review research report 2 [Online]. London: BMA; 2002. Available from: URL: http://www.bma.org.uk/ap.nsf/Content/Healthcare+funding+ review+research+report+2.
4.
European Observatory on Health Care Systems. Health Care Systems in Transition: United Kingdom [Online]. London: European Observatory on Health Care Systems; 1999. Available from: URL: http://www.euro.who.int/document/e68283.pdf.
5.
Healthcare Commission [Online].. Available from: URL: http://www.healthcarecommission.org.uk/Homepage.cfm.
6.
Commission for Social Care Inspection [Online].. Available from: URL: http://www.csci.org.uk/.
7.
Whoriskey M. Developing quality and standards.In: Brown M, editor. Learning disabilities: a handbook of integrated care. Salisbury: APS; 2003. p. 22-9.
8.
SWIA. Performance Inspection Handbook [Online]. Edinburgh: Social Work Inspection Agency; 2004. Available from: URL: http://www.swia.gov.uk/.
9.
Beswick J. An evaluation of the effects on quality of life outcome measures for people with learning difficulties (mental handicap) of changes in the living situation from hospitals to community environments. Unpublished PhD thesis. University of Manchester; 1992.
10.
Schalock RL. The concept of quality of life: what we know and do not know. Journal of Intellectual Disability Research 2004; 48(3):203-16.
11.
Cummins RA. Moving from the quality of life concept to a theory. Journal of Intellectual Disability Research 2005; 49(10):699-706.
12.
HMIe Services for Children Unit. A common approach to inspecting services for children and young people; consultation document October 2005. Edinburgh: HMSO; 2005. Also available from: URL: http://www.hmie.gov.uk/documents/publication/caiscyp.pdf.
13.
Baum NT. The multi-focal approach and quality of life content in the treatment of people with dual diagnosis.In: Keith K, Schalock R, editors. Cross-cultural perspectives on quality of life. Washington DC: American Association on Mental Retardation; 2000. p. 93-103.
14.
Baum NT. Quality of life is not only for people served—it is also for staff: the multi-focal approach. Journal of Intellectual Disability Research 2005; 49(pt 10):809-11.
15.
Schalock RL, Verdugo MA. Handbook on quality of life for human service practitioners. Washington: American Association on Mental Retardation; 2002.
16.
Hatton C, Emerson E, Lobb C. Developing outcome-based performance indicators for adults with learning disabilities: a proposed framework and possible outcome indicators draft 1. Lancaster: Institute for Health Research, Lancaster University; 2005. In press.
17.
Perry J, Felce D. Correlation between subjective and objective measures of outcome in staffed community housing. Journal of Intellectual Disability Research 2005; 49(4):278-87.
18.
Jacobson NS, Truax P. Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting Clinical Psychology 1991; 59:12-9.
19.
Jenkins R, Brooksbank D, Miller E. Ageing in learning difficulties: the development of health care outcome indicators. Journal of Intellectual Disability Research 1994 Jun; 38(3):257-64.
20.
Kazdin AE. The meanings and measurement of clinical significance. Journal of Consulting and Clinical Psychology 1999; 67:332-9.
21.
Glendinning C. Research to Develop New Approaches to Measuring and Understanding Social Services Outputs and Productivity: Comments from Social Policy Research Unit, University of York, 25 March 2004 [Online]. York: Social Policy Research Unit, University of York; 2004 Dec. (DHP 2053). Available from: URL: http://www.york.ac.uk/inst/spru/research/pdf/pssresearch.pdf.
22.
Hatton C, Emerson E, Lobb C. Evaluating the impact of valuing people. Report of phase 1: a review of existing national datasets. Lancaster: Institute for Health Research, Lancaster University; 2005.
23.
Elliott J, Hatton C, Emerson E. The health of people with learning disabilities in the UK: Evidence and implications for the NHS. Journal of Integrated Care 2003; 11(3):9-17.
24.
The Council on Quality and Leadership in Supports for People with Disabilities: Personal Outcomes Chart Book. Towson, MD: The National Centre on Outcomes Research; 1999. Also available from: URL: http://eric.ed.gov/.
25.
Verdugo MA, Schalock RL, Keith KD, Stancliffe RJ. Quality of life and its measurement: important principles and guidelines. Journal of Intellectual Disability Research 2005; 49(10):707-17.
26.
De Waele I, van Loon J, Van Hove G, Schalock RL. Quality of life versus quality of care: implications for people and programs. Journal of Policy and Practice in Intellectual Disabilities 2005; 2(3–4):229-39.
27.
Barkham M, Mellor-Clark J. Rigour and relevance: the role of practice-based evidence in the psychological therapies.In: Rowland N, Goss S, editors. Evidence-based mental health. London: Routledge; 2000. p. 127-44.
28.
Barkham M, Margison F, Leach C, Lucock M, Mellor-Clark J, Evans C, et al. Service profiling and outcomes benchmarking using the CORE-OM: towards practice-based evidence in the psychological therapies. Journal of Consulting and Clinical Psychology 2001; 69:184-96.
29.
Gardner JF, Carran DT. Attainment of personal outcomes by people with developmental disabilities. Mental Retardation 2005 Jun; 43(3):157-74.
30.
Roy A, Matthews H, Clifford P, Fowler V, Martin DM. Health of the Nation Outcome Scales for People with Learning Disabilities (HoNOS-LD). British Journal of Psychiatry 2002 Jan; 180:61-6.
31.
The Council on Quality and Leadership. The personal outcome measures. Maryland: CQL; 2000.
32.
Fitzpatrick R, Davey C, Buxton MJ, Jones DR. Evaluating patient-based outcome measures for use in clinical trials. Health Technology Assessment 1998; 2(14):12-4.
33.
Slade M, Thornicroft G, Glover G. The feasibility of routine outcome measures in mental health. Social Psychiatry and Psychiatric Epidemiology 1999; 34:243-9.
34.
Berger M. Outcomes and effectiveness in clinical practice. Leicester: British Psychological Society; 1996.
35.
Stufflebeam DL, Madaus GF, Kellaghan T. Evaluation models: viewpoints on educational and human services evaluation New York: Kluwer Academic Publishers; 2000.
36.
Walter G, Cleary M, Rey J. Attitudes of mental health personnel towards rating outcome. Journal of Quality in Clinical Practice 1998; 18:109-11.
Figures
Figure 1

Joint Inspection Bingo.

Figure 2

Quality of Life.

Figure 3

Quality Outcome Indicator.

Figure 4

The collection and verification of outcomes by triangulation.

Figure 5

Categories to describe performance.