From community informed outcomes framework to routine & systematic outcome measurement

Measuring What Matters to Communities: What you measure matters to systems

Catherine Godward, Bromley by Bow’s Research and Evaluation Manager, and implementation lead for Evidence Into Practice, stage two of the Unleashing Healthy Communities project in Bromley by Bow, shares the journey Bromley by Bow has taken to measure what matters to communities. 

Contact: insights@bbbc.org.uk

Download the PDF here

Introduction:

In June 2020, at the start of the pandemic, and with funding from the Health Foundation, Bromley by Bow tasked itself with translating its co-produced Community Informed Outcomes Framework into a measurable tool.

The Framework, co-produced through Unleashing Healthy Communities summarises Tower Hamlets’ people’s answers to the question ‘what are the ingredients of a good life?’, the answers to which have been grouped into six outcomes, (illustrated here in the Bromley by Bow logic model):

• Basic needs met
• Built knowledge and skills
• Connected to others
• Connected to place and community
• Contributed to the community
• Strengthened personal resources

Each outcome can stretch along a continuum e.g. someone may feel motivated to help others and someone may lead a volunteer group. All outcomes are intended outcomes of the combined services delivered at Bromley by Bow.

As an organisation we have turned the Framework into a quantitative method to routinely and systematically capture client outcomes, and this is our journey….

The setting:

Bromley by Bow’s integrated health and community services, housed in a Community Centre and three General Practices, takes a holistic, people-first approach to support health creation, drawing on assets within the local community in Tower Hamlets. Our community has complex needs, already suffering deprivation for many years and the Covid-19 pandemic has amplified inequalities.

Why:

With resources tight everywhere, now, more than ever, it is crucial that Bromley by Bow has access to outcome measurement data that truly reflects change due to the health creation services it delivers, based on what’s important to members of the community, so that we can:

• Share results with those to whom we are accountable: clients, the local community, funders, partners, stakeholders, and internal colleagues.
• Carefully match specific services and the outcomes generated with specific clients that would benefit.
• Channel the clients’ voices in programme design and delivery.
• Identify what works and in what way, and what doesn’t, sharing lessons learned.
• Focus on intended outcomes that the community truly wants.
• Create a basis for dialogue with stakeholders about how to work effectively with the community to maximise opportunity and fill gaps.

Steps taken so far:

Coordinated by Bromley by Bow Insights this work has included the participation of the whole organisation and three major, iterative, steps:

• Scoping through desk research and 1:1 interviews with internal and external colleagues to understand the need and potential use for outcome data and possible approaches that have been, and could be taken.
• We delivered a ‘100-day challenge’ which included:
  1. A series of action focused discussions with all service delivery staff that:
     1. Identified intended outcomes of their work for clients, in such a way that the outcome would be recognisable to clients.
     2. Tested out possible methods with members of the team.
  2. Team discussions to reflect and select indicators (i.e. questions) that best capture intended outcomes for clients.
• Data collection testing via cognitive interviews and trial data collection by the Social Welfare Advice and Rapid Response Social Prescribing services.

At each step of the way the work was supported via an internal and an external advisory group and two weekly meetings with two members of the Executive Leadership Group. The focus of each group was slightly different and covered how to embed within the existing service with minimal disruption, and technical and academic rigour. It became clear early on that organisational outcome measurement = organisational change and influence at a senior level made that possible.

Figure 1 The journey so far, a timeline

The result:

Bromley by Bow’s new outcome measurement approach generates data consistently from all clients across all services. It includes a tool with 13 domains, 41 bespoke/validated indicators, 36 subjective, asked at the close of an engagement, and both ONS4 and MYCAW, two pre/post measures of wellbeing. Both were chosen because the component parts are all intended outcomes of Bromley by Bow’s services.

Table 1 Bromley by Bow six stretch outcomes and domains

Services select relevant outcomes, domains and indicators early on for each activity delivered. Outcome measurement data capture is predominantly implemented via a 1:1 interview as part of the cycle of usual client contact by Bromley by Bow practitioners and service delivery colleagues, at three key stages:

1. When a client first engages with Bromley by Bow, data that captures demographics, presenting need and the ONS4/MYCAW baseline are captured and stored in central data systems.
2. Bromley by Bow staff code and record presenting need to intended stretch outcome(s) to monitor need against the Outcomes Framework.
3. Change is captured at the close of a service taken up. In the case of social prescribing the close of the service is both after one engagement and after all engagements.

Where possible we supplement with qualitative data gathered from:

• in-depth 1:1 interviews with clients, where possible, to understand and record any changes that have occurred due to the programme, set within a narrative.
• team reflection workshops during programme implementation to:
  • capture evidence of what works well, what isn’t working well,
  • assess improvements that have taken place and develop further improvements for ‘real-time’ implementation that supports the delivery of intended and agreed outcomes.

Challenges along the way:

This hasn’t been your ‘usual’ outcome measurement study and we have had to overcome challenges along the way, summarised in the table below. Some may be particular to the Bromley by Bow Centre but many will be common to other organisations undertaking a similar task.

Table 2 Challenges faced and overcome

Changes so far:

• Outcomes in the Framework are now a key driver of all Bromley by Bow’s work and maps to every activity.
• One member of the delivery team said outcome measurement is supporting teams to focus on the people within the community.
• A practitioner reported that the 100-day challenge immediately affected thinking, planning and action how best to deliver their activity/with new focussed intended outcomes. The session provided the bare bones of a logic model.
• A senior staff member said the process had provided a sense of team work – ‘measuring together and working together’.
• Outcomes have helped teams to look beyond the boundaries of their specific activities to take a person specific rather than project specific approach, connecting more with other practitioners and teams.
• Practitioners have reported that measuring changes in client mental health has encouraged them to become more aware of their own mental health and adjusted their own working life accordingly e.g. taking time out of the working day to do something else and making sure time is made to take a break for lunch.
• Some colleagues feel that the development work has helped colleagues to explain better and feel more secure now about the work we are doing and why.

 

What we have learned:

There has been a lot to learn since embarking on this process, here are five key things:

1. Clients are pleasantly surprised to be asked about changes. This comes from feedback from the cognitive interviews and testing. That being said, clients need to see a clear link between the questions and them.
2. Whole organisation outcome measurement = whole organisation change

The process has required practitioners to be open to:

• changing how they spend time with clients;
• changing the approach to client engagement;
• thinking more about the collective purpose of the Bromley by Bow Centre and not just individual activities delivered.

A dedicated post holder has helped to facilitate the change. Whole organisation change takes time, be patient!

3. Engage colleagues every step of the way and build ownership

Organisational change has been possible through whole organisation ownership of the change. Through continual profile raising and momentum created by the 100-day challenge, team members have been able to participate in every stage. Practitioners, who know the clients best, were immersed in development which led to the delivery of an outcome measurement webinar co-produced by practitioners who led breakout groups with external practitioners and researchers about each stage of development and testing.

Regular engagement with and action by senior representatives from across Bromley by Bow has been essential to providing the perspective of the wider context and opportunities and barriers to change, and opportunities for action.

4. Work with the status quo

As far as is possible the approach we have developed has been embedded within existing systems and procedures. These are designed to enable practitioners to support and help clients and clients are the priority in everything we do so maintaining these has been key.

Added to which practitioners are stretched and necessarily focused on generating immediate outcomes for clients and not measurement. It has been important to recognise these pressures and tipping points and work together rather than imposing a new approach.

5. Well selected internal and external advisors bring a combination of positioning, independence, credibility and practical support. 

What next:

• We are rolling out to every service, co-producing service specific approaches to embed questions into conversations.
• Data dashboards have been produced and we will co-produce more.
• We will be applying the new approach to every relevant funding agreement.
• We will be looking for funding to provide the investment security to maintain outcome measurement in this way at Bromley by Bow.

In many ways, as is often the case, our journey is just beginning.