Evaluating projects and events

Printer-friendly versionPrinter-friendly version

Evaluating and reflecting on the successes of your projects and events is a useful exercise both during delivery and after completion. To measure success, you will need to establish a baseline and collect data and feedback from those who have been involved. The results and insights gained can then be used to improve your engagement methods, revise your project plan and to identify successes to communicate to your community and more widely.

Click on the ideas below for practical suggestions and detailed case studies.

Establishing a baseline and defining success

Before you are able to evaluate the success of your projects or events, you need to establish a baseline and define the meaning of “success”. Without taking these first steps it will be difficult to measure or identify your achievements. This can be an important part of your initial visioning exercises when designing your project concept.

Establishing a baseline. The baseline is the position before you have taken any action. Knowing this allows you to compare the situation before and after your project or event in order to determine what the impact has been. The baseline will depend on what you want to measure and could include, for example, the number of people on your mailing list before an event, the number of homes with a certain energy efficiency installation or the initial carbon footprint or energy consumption of a household or group of households before your project starts. The data to determine your baseline could be collected via some of the Collecting data and feedback suggestions.

Defining success. The meaning of “success” will vary depending on the objectives set for the particular project or event. Having clear objectives is well worthwhile in order to have direction, focus and something to work towards. Objectives could relate to, for example, the level of participation, awareness and engagement, completion of phases of a construction project, financial returns generated or outcomes in terms of carbon emission reductions or energy savings. Often the best way to define success is to set targets from the outset. Targets can vary in specificity, although making them SMART (specific, measurable, achievable, relevant and time-bound) will make it easier to assess your progress.

Case Studies
Country: UK

Ashton Hayes carried out a baseline survey of CO2 emissions from their village in order to be able to assess their progress towards their aim of becoming England’s first carbon neutral village. The baseline survey was conducted in May 2006 and took about 4-6 weeks. Each year a team from the University of Chester repeats the survey to calculate the village’s success at cutting emissions. Students from the University carry our door-to-door surveys to ask residents questions about their lifestyles, including their domestic energy use, travel and holidays. Using this data, they compute their domestic footprint in order to ascertain that of the village as a whole. The calculations are based on certain assumptions established at the beginning of the project relating to, for example, eligible activities which can contribute towards reducing carbon footprints, average housing characteristics, and efficiency levels of vehicles, white goods, boilers and other equipment. The results from the first 5 surveys demonstrated that the community has managed to cut its carbon emissions by 23% through behavioural changes such as switching off appliances and changing to low energy light bulbs. Residents also provide feedback on the surveys and guidance on what further actions they can take to cut their carbon emissions.

Country: UK

Resilient Energy Great Dunkiln (REGD) is a joint venture between The Resilience Centre, a landowner and the community of St Briavels. Together they raised £1.4 million in five months to install a 500kW wind turbine in 2012 in the parish of St Briavels.

Various targets to measure the success of the project were determined  by a number of factors, namely an upfront independent energy generation report, ongoing monitoring, open book accounts (publically available), visitor numbers to site events, visitor numbers to the farm in between events, comments from local institutions and their keenness to support other similar projects in neighbouring parishes. 

Performance is measured on the successful delivery of projects funded by the community donations and level of support provided by the St Briavels community. REGD also measure the quality and value of the projects being proposed and how they meet future needs and increase the resilience of the community through reducing CO2 emissions, expanding community owned assets and additional self funded projects.

Published information is presented at annual District Council and Parish Council meetings. This information is used to demonstrate how monies are being spent and to encourage the community to think more strategically about its future needs.  REGD also publish twice yearly articles and blog posts in various magazines and newsletters to further engage the community. Many letters of support and thanks have been sent from St Briavels residents while many others volunteer their time to support this and other projects.

Country: Belgium

Interleuven works with local municipalities to deliver thermal imaging projects. Taking thermal images, or infrared photos, of homes before installing insulation or other means of reducing heat loss provides a useful record of baseline energy performance. Thermal images are used to provide a very visual presentation of baseline energy performance to residents, highlighting where and how much energy is being lost from their home. Residents are given expert advice on the various options available to reduce this energy loss highlighted and guidance on possible subsidies. If a resident orders some insulation works, a new thermal image is taken after these changes have been made, which is then compared with the baseline image in order to demostrate the reduced heat loss.

Collecting data and feedback

The most appropriate method for collecting data and feedback will depend on the type of data to be collected, from who and for what purpose. In some cases it may be easiest or only necessary to keep simple records as you go along.

Remember to use the data and feedback collected and compare it with your baseline and against your objectives to inform your project and related activities. Largely this comes down to communicating your achievements with your supporters and the wider community, establishing an evidence base and using the information gathered to learn from the experience and to improve your approach.

Methods for collecting data and feedback include:

Surveys and questionnaires. Surveys and questionnaires can be a quick and easy way to collect feedback from a range of stakeholders. Topics could include feedback on a meeting or event, information about energy use or behaviours of households, or suggestions and support for a project idea. Online surveys produced using platforms such as SurveyMonkey and distributed to your mailing list can enable you to collect responses in a clear and common format from a large number of people. Alternatively, paper surveys can be handed out at public meetings or events for people to fill in and return before they leave.

Designing your surveys to be short and to the point will make it more likely that people will fill them out. Multiple choice questions can help to ensure the feedback you gain is comparable and in a common format, avoiding problems of misinterpretation and also being quicker and easier for people to answer. If you are able to provide incentives for filling out surveys, such as entry into a prize draw, this can help to maximise your response rate.

Focus groups. Focus groups can be a great way to test and trial ideas and initiatives, and to gauge local opinions on certain issues. Compared with surveys and questionnaires, focus groups tend to produce broad, open-ended qualitative results, communicated both verbally and non-verbally. This can enable your results to capture nuances, giving you a better idea of what people really think and feel about a topic. Focus groups can either be one-off, for example to inform the design of an initiative before it is delivered or after delivery to assess its success, or both before and after to compare how expectations and views changed.

For your results to be most useful, make sure your focus group is representative of the whole community or the sub-group within it that you are seeking feedback from. On the day, ensure a relaxed, informal and non-confrontational atmosphere in which participants feel able to be open and honest about their thoughts and feelings.

Keeping records. Often the easiest way to collect data on the impact of your project is simply through keeping records of your activities. This could include anything from recording the number of people who attend your events, volunteer their time or sign up to your mailing list, to the number of installations you have delivered, the amount of energy generated or saved, or the money coming into your project. The better your records, the better the information available to you, the better your decisions, and the easy it will be to make a persuasive case for your project.

Informal communications. Simply talking with people and asking them for feedback in an informal setting can help to assess what is going well and where there are areas for improvement. Conversations with new and different people at your public meetings and events, or people you bump into in the street or at the shops, can lead to new and useful insights that might not otherwise have been recognised or considered.

Case Studies
Country: Belgium

Students at the University of Leuven organised the 'Cité Climate Challenge', a contest which involved 820 students from 14 residential living blocks in Arenberg competing against each other to save the most water, gas and electricity. The contest put data collection and reporting at the heart of the project, drawing students' attention to their water and energy consumption and highlighting the impact that simple changes in behaviour can have. The data collected demonstrated that 4.5 tonnes of CO2 was saved through the project, and the winning residential block, block 3, was awarded €1,000.

Country: UK

Stroudco Food Hub collaborated with researchers from Cardiff University to run a focus group at a local school to gain an understanding of local food shopping habits. They asked a group of local mums to explore their food-buying habits using a mapping game and discussed with them the pros and cons of local food. They then explained their Food Hub initiative and its aims, and invited recommendations about how they could make their offer most appealing. Suggestions included being able to buy fruit and veg frequently and in small quantities, with more affordable options and special offers. It was suggested that the Food Hub should focus on cheap items (e.g.  potatoes and carrots), staple items (e.g. milk, bread, butter, cheese and eggs) and popular meat products (e.g. chicken and beef mince). The discussions also led to options being explored for more flexible arrangements for picking up produce, such as through liaising with the local family centre and schools to set up pick up points.

Country: UK

Low Carbon West Oxford has carried out consultations and distributed questionnaires to reveal barriers to participation in their projects. This research revealed time and cost barriers experienced by many members of the community. In order to overcome these barriers, the group developed small grant schemes for households to install carbon saving measures and to purchase childcare to allow them to attend workshops. They also developed more flexible opportunities to enable greater involvement of time-pressed individuals, for example through offering one-off opportunities and home visits. Surveys of participating households revealed a wide range of motivations for getting involved, which was used to tailor messages and communications for different groups.

Source: Low Carbon West Oxford and West Oxford Community Renewables (2010), Low Carbon Living: Power to make it possible

Country: UK

Bath and West Community Energy calculate forecasts for the output of their solar PV projects and track the actual output produced. These figures are calculated and communicated quarterly and posted on their website in the form of a simple graph. The graph shows the performance of all their operating solar PV projects since commissioning, comparing actual and forecast output. The line shows the percentage to which all the projects have performed above or below forecast when considered cumulatively since the date they were commissioned. This has enabled them to identify and communicate that overall their projects have performed 5% above forecast on average since commissioning. This was as high as over 10% above forecast in March 2012. They also have separate webpages showing the output from each of their individual projects, with graphs and explanations. This approach enables them to have evidence of the benefits of their projects which can be communicated to the community to engage both new and existing supporters.