Measuring your outcomes
Lewisham Youth Theatre
Lewisham Youth Theatre (LYT) runs participatory drama projects for young people who live or learn in Lewisham. It reaches over 150 people through its drama programmes and about 700 young people through its outreach.
They work with 8 – 24-year-olds, and receive referrals from partners across Lewisham within different schools, CAMHS, Baseline, etc. Each of their sustained projects runs between six weeks and five months, with one cohort taking part in weekly sessions over that period of time.
They offer an inclusive service, targeting young people in need. At least 70 per cent identify as having specific needs, ranging from SENs, young carers, those involved with social services, young parents etc. Although they recruit and target young people in need, when in sessions everyone is treated equally.
Victoria Shashkan (pictured) has been the Executive Director of LYT since 2011. She has overseen the development of a Theory of Change and a programme of monitoring and evaluation across the organisation, which has resulted in a more efficient system and increased funding.
Our overall goal is that everyone who interacts with us has improved wellbeing and life-chances. We use theatre as an intervention to achieve this goal.
We developed a Theory of Change – what we call our Organisational Outcomes & Targets - to help us articulate our overall goal and outcomes, and how we hoped to achieve this with our targets.
For each project we run, we measure the development of participants’ social and emotional skills, their creative skills, their leadership and ambition. With a service like this, you’re never going to prove a causal link, but we try to get as much evidence as we can of the difference we make in the way we monitor, who we monitor and what we monitor.
We’ve developed our fundraising strategy over the last five years to look much more strategically about applying for funding that is for the programme rather than from project to project.
This was prompted, in part, by funding the youth theatre received from Big Lottery Fund and Children in Need, which made us look at our outcomes and monitoring across our projects.
We think about it now internally as a whole programme of work that we need to deliver in order to meet our outcomes, rather than individual strands of work.
The first thing we do is to keep registers of all of our participants. We use Substance Views database, which allows us to track our users across projects. It’s great that we got introduced to this via the Council, who invested in it for youth service commissioning. It’s built with participatory non-profits in mind.
What that allows us to do is to see the percentage of young people who finish their projects and who participate in three or more sessions and our retention rates (85% finish our programme). The administrator updates the database weekly.
At the beginning of the project, participants fill in a baseline questionnaire that asks them about their social and emotional skills, and what they want to get out of the project. Their engagement is monitored throughout the project through tutor-completed sessional evaluations.
The goal of all our evaluation methods is that we use them as a tool that helps young people develop their skills. So what we’d often do – depending on the age groups – is use a document that lists what, say, ‘confidence’ might look like. We don’t just give a tick-box form out to the participants; we use it in an interactive way.
At the end of the project we have an evaluation session and get people thinking about how far they’ve come. So, we’re not asking the same questions and expecting that to tell us the distance that they’ve travelled. We ask them ‘How much have you improved in this skill?’ They’re telling us the distance they’ve travelled and we’re recording what they’ve told us. Then we’re able to say, ‘X per cent report that they have significantly improved in this area’.
We also get quotes and narrative to prove they’ve gained those skills, as well as feedback from teachers, referral partners, audience feedback forms etc. Then they take all of that information and project leads compile the data into a report that evidences against performance targets from their Theory of Change.
We have two types of evaluation:
Evaluation used to evidence impact of the work on service users, which is important for funders and the organisation to know you’re making a difference.
- Critical evaluation that isn’t necessarily for funders but for internal evaluation. At the beginning of each project we look at recommendations from the project before planning for the next one, including staff evaluation of what went well. That’s a really important process for the development of work to respond to the needs of users. And because it’s the same age group every year we’re also able to replicate and improve it.
We’re planning a major evaluation of our programmes during 2016 to evaluate the long-term impact LYT has on the wellbeing and life-chances of children and young people that take part with us.
I’ve given up on the idea that it will be statistically relevant, but it will help us to get more information on the long-term impact we have and to set our agenda for the next 5 years. It’s also a way of getting young people involved in running the projects and thinking about how they make a difference.
We’ll be sending out an online survey for anyone who has ever participated in LYT, and particularly targeting those who have used our services in the last four years, asking for their opinions on how LYT impacted them, what they’re doing now, their work and education prospects, etc.
Our Members’ Committee has helped develop a survey and will be running focus groups in the Autumn. We’ll follow-up on case studies we’ve done over the last 4 years, and we’ll also look at literature that links to the development of skills and the link to improved wellbeing.
Victoria has identified three key factors for success:
Get help to define your outcomes: ‘What we recognised was that we were really good at collecting information but not that great at using it to communicate our impact. We got advice from an independent evaluator to help us structure a pyramid of evidence. We looked at targets from all our funders to look at what we needed to monitor. Then we said, “These are always going to be our outcomes”.’
Embed an ethos of evaluation in the organisation: ‘We wanted to find a way that everyone was invested in collecting information. By putting the onus on the sessional workers and getting them on board, we found that two things happen: firstly, they do it in a way that is useful for them and the development of their programmes, and secondly they actually collate the information rather than my collating it at the end of each project. The ethos of evaluation has been embedded in the organisation, which is really important.’
- Involve your service users: ‘We used our Members Committee when we developed our forms and tried to keep them as standardised as possible. Members responded to the wording. This also helps them reflect on how the projects help them.’
Here are some useful resources from Lewisham Youth Theatre that you may find helpful
This case study is part of the Show your impact programme, funded by City Bridge Trust.